METHOD, APPARATUS, AND PROGRAM FOR MOVING IMAGE SYNTHESIS 



BACKGROUND OF THE INVENTION 
Field of the Invention 
The present invention relates to a method and apparatus 
for synthesizing a plurality of frames, sampled from a moving 
image, to obtain a synthesized frame having a higher resolution 
than the sampled frames. The present invention also relates w 
to a program that causes a computer to execute the method for 
synthesizing moving images. 

Description of the Related Art 
With the recent spread of digital video cameras, it has 
become possible to handle moving images in units of single 
frames thereof. When these frames of the moving image are to 
be output as prints, it is necessary to heighten their 
resolutions, to improve the image qualities thereof. 
Therefore, methods for sampling a plurality of frames from a 
moving image, then generating a single synthesized frame having 
higher resolution than the plurality of frames, have been 
proposed (for example, in Japanese Unexamined Patent 
Publication No. 2000-354244) . These methods derive motion 
vectors among the plurality of sampled frames. Then, the 
methods calculate signal values to be inserted among pixels, 
based on the motion vectors, when generating the synthesized 
image from the plurality of frames. In the method disclosed 
in Japanese Unexamined Patent Publication No . 2000-354244, each 
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of the sampled frames is divided into a plurality of blocks. 
Then, perpendicular coordinate coefficients that correspond 
among the frames are calculated. Thereafter, data regarding 
high frequency components within the perpendicular coordinate 
5 coefficients are synthesized with low frequency components 
within other blocks, to calculate pixel values to be inserted. 
Therefore, necessary data is not reduced, and a synthesized 
frame having high image quality can be obtained. In addition, 
this method calculates motion vectors at resolutions finer than 

10 the distances between pixels. Therefore, movement between 
frames is accurately compensated for, enabling the obtainment 
of synthesized frames having still higher image quality. 

Another method for synthesizing frames of a moving image 
has been proposed, as disclosed in: Y. Nakazawa, T. Komatsu, 

15 and T. Saito, "Obtainment of Finely Detailed Digital Images by 
Integrating Frames", Journal of the Television Academic Society, 
1995, Vol. 49, No. 3 p299-308. This method designates one of 
a plurality of frames as a reference frame. A reference patch, 
having a plurality of rectangular regions therein, is provided 

20 in the reference frame. A second patch, which is the same as 
the reference patch, is provided in a second frame other than 
the reference frame. The second patch is moved and/or deformed 
so that the image within the second patch matches the image 
within the reference patch. Then, correspondent relationships 

25 between pixels within the second patch and pixels within the 
reference patch are estimated, based on the moved and/or 
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deformed second patch and the reference patch. Thereby, a 
plurality of frames are accurately synthesized. 

In the method disclosed by Nakazawa et al . , correspondent 
relationships among the reference frame and the other frames 
5 are estimated. Following this estimation, the reference frame 
and the other frames are assigned to an integrated image, having 
a necessary resolution. Thereby, a finely detailed 
synthesized frame is obtained. 

There are cases in which a subject included in a frame, 

10 or included in a local portion of a frame, moves greatly or at 
extremely high speed among frames. In these cases, movement 
and/or deformation the patches in the method disclosed by 
Nakazawa et al . may not be able to follow such movement. In 
the case that the movement and/or deformation of the patches 

15 cannot follow the movement and/or deformation of the subject, 
the entirety of a synthesized frame may be blurred, or the 
subject therein, which exhibits great movement, may be blurred. 
Thus, there is a problem that a synthesized frame having high 
image quality cannot be obtained. In these cases, the movement 

20 and/or deformation of the patches can be made to follow the 
movement and/or deformation of the subject, by dividing the 
patches into a greater number of rectangular regions. However, 
if the number of rectangular regions is excessively increased, 
then each of the rectangular regions becomes extremely small, 

25 thereby causing great difficulty in following the movement 
and/or deformation of the subject. 
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SUMMARY OF THE INVENTION 
The present invention has been developed in view of the 
circumstances described above. It is an object of the present 
invention to enable obtainment of synthesized frames having 
5 high image quality, employing an optimal number of rectangular 
regions within patches. 

The first method for synthesizing moving images comprises 
the steps of: 

sampling two consecutive frames from a moving image; 
10 providing a reference patch having at least one 

rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
reference patch, in the other of the two frames; 
15 moving and/or deforming the second patch in the other of 

the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
20 the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
25 space of the reference frame, based on the correspondent 
relationships; 



calculating a correlative value, which represents 
the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
converted frame; 

5 varying the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 

10 thereby obtaining a plurality of correlative values 
corresponding to the number of rectangular regions within the 
reference patch and the second patch; 

comparing the degrees of correlation for each number of 
rectangular regions; and 

15 generating a synthesized frame having a higher resolution 

than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
reference patch, based on the correspondent relationships, 
which were estimated among the pixels of the reference patch 

2 0 and the second patch having the number of rectangular regions 
therein that yielded the highest degree of correlation. 

Here, "varying the number of rectangular regions... in 
a stepwise manner" refers both to a case in which the number 
of rectangular regions is gradually increased from a small 

25 number to a predetermined number in a stepwise manner, and to 
a case in which the number of rectangular regions is gradually 
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decreased from a large number to a predetermined number in a 
stepwise manner. 

Here, the "correlative value" may be calculated for each 
rectangular region within the patches of the coordinate 
5 converted frame and the reference patch of the reference frame. 
Alternatively, the correlative value may be calculated for the 
entire frames. 

Here, "comparing the degrees of correlation" refers to 
comparing the correlative values between frames, in the case 

10 that the correlative values have been calculated for the entire 
frames. In the case that the correlative values have been 
calculated for each of the rectangular regions, "comparing the 
degrees of correlation" refers to comparing the correlative 
values among each of the rectangular regions. Note that the 

15 comparison is performed between correlative values which have 
been obtained between patches having different numbers of 
rectangular regions therein. As the number of rectangular 
regions increase, the sizes thereof decrease. Therefore, 
comparisons between correlative values, which have been 

20 calculated for a certain number of rectangular regions, and 
those which have been calculated for a different number of 
rectangular regions, are performed such that the correlative 
values of the larger sized rectangular regions are compared with 
the mean or median value of a plurality of the smaller sized 

25 rectangular regions. 

Note that in the first method for synthesizing moving 
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images of the present invention, the correspondent 
relationships employed in the generation of the synthesized 
frame may be those which have been estimated for each 
rectangular region . 
5 In addition, in the first method for synthesizing moving 

images of the present invention, at least one component that 
constitutes the frames may be employed during the estimation 
of the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
10 values, and the generation of the synthesized frame. 

The second method for synthesizing moving images of the 
present invention comprises the steps of: 

sampling at least three consecutive frames from a moving 

image; 

15 providing a reference patch having at least one 

rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

providing a second patch, which is the same as the 
20 reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
within the reference image; 

estimating correspondent relationships among the pixels 
25 within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
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patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships ; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame; 

varying the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
thereby obtaining a plurality of correlative values 
corresponding to the number of rectangular regions within the 
reference patch and the second patch; 

comparing the degrees of correlation for each number of 
rectangular regions ; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 
images within the second patch and the reference patch, based 
on the correspondent relationships, which were estimated among 
the pixels of the reference patch and the second patch having 
the number of rectangular regions therein that yielded the 
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highest degree of correlation; 

obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
5 plurality of correlative values, and comparing the degrees of 
correlation, for all of the plurality of sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

Note that in the second method for synthesizing moving 
10 images of the present invention, the correspondent 
relationships may be estimated for each of the rectangular 
regions . 

In addition, in the second method for synthesizing moving 
images of the present invention, at least one component that 

15 constitutes the frames may be employed during the estimation 
of the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
values, the generation of the intermediate synthesized frames, 
and the generation of the synthesized frame. 

20 The "at least one component that constitutes the frames" 

refers to at least one of three color components RGB, in the 
case that the frames are constituted by RGB color data, for 
example. Alternatively, in the case that the frames are 
constituted by luminance chrominance components YCC, the "at 

25 least one component" refers to at least one of the luminance 
and chrominance components, and preferably the luminance 
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component . 

The first moving image synthesizing apparatus of the 
present invention comprises: 

a sampling means, for sampling two consecutive frames 
5 from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
10 same as the reference patch, in the other of the two frames; 
moving and/or deforming the second patch in the other of the 
two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
15 other frame and the pixels within the reference patch of the 
reference frame, based on the second patch after the movement 
and/or deformation thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
20 second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
25 frame and the coordinate converted frame; 

a control means, for controlling the correspondent 



10 



relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to vary the number 
of rectangular regions within the reference patch and the second 
patch in a stepwise manner, to estimate the aforementioned 
5 correspondent relationships, to obtain the coordinate 
converted frames, and to calculate the correlative values for 
each number of rectangular regions, thereby obtaining a 
plurality of correlative values corresponding to the number of 
rectangular regions within the reference patch and the second 
10 patch; 

a comparing means, for comparing the degrees of 
correlation for each number of rectangular regions; and 

a synthesizing means, for generating a synthesized frame 
having a higher resolution than either of the two frames, by 

15 administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the highest 

20 degree of correlation. 

Note that in the first moving image synthesizing 
apparatus of the present invention, the correspondent 
relationships employed in the generation of the generation of 
the synthesized frame may be estimated for each rectangular 

25 region. 

In addition, in the first moving image synthesizing 
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apparatus of the present invention, the correspondent 
relationship estimating means, the coordinate converting means, 
the correlative value calculating means, and the synthesizing 
means may employ at least one component that constitutes the 
5 frames during the estimation of the correspondent relationships, 
the obtainment of the coordinate converted frames, the 
calculation of the correlative values, and the generation of 
the synthesized frame. 

The second moving image synthesizing apparatus of the 

10 present invention comprises: 

a sampling means, for sampling at least three consecutive 
frames from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 

15 region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 
patch, in another of the plurality of sampled frames; moving 
and/or deforming the second patch in the other frame so that 

20 an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 
the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 

25 thereof and the reference patch; 

a coordinate converting means , for obtaining a coordinate 
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converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
5 a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 

10 and the correlative value calculating means, to vary the number 
of rectangular regions within the reference patch and the second 
patch in a stepwise manner, to estimate the aforementioned 
correspondent relationships, to obtain the coordinate 
converted frames, and to calculate the correlative values for 

15 each number of rectangular regions, thereby obtaining a 
plurality of correlative values corresponding to the number of 
rectangular regions within the reference patch and the second 
patch; 

a comparing means, for comparing the degrees of 
20 correlation for each number of rectangular regions; and 

a synthesizing means, for: obtaining an intermediate 
synthesized frame having a higher resolution than either the 
reference frame of the other frame, by administering 
interpolation calculations on the images within the second 
25 patch and the reference patch, based on the correspondent 
relationships, which were estimated among the pixels of the 
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reference patch and the second patch having the number of 
rectangular regions therein that yielded the highest degree of 
correlation; obtaining a plurality of intermediate synthesized 
frames by estimating the aforementioned correspondent 
5 relationships, obtaining the coordinate converted frames, 
calculating the plurality of correlative values, and comparing 
the degrees of correlation, for all of the plurality of sampled 
frames; and generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 
10 Note that in the second moving image synthesizing 

apparatus of the present invention, the correspondent 
relationships employed in the generation of the generation of 
the synthesized frame may be estimated for each rectangular 
region . 

15 In addition, in the second moving image synthesizing 

apparatus of the present invention, the correspondent 
relationship estimating means, the coordinate converting means, 
the correlative value calculating means, and the synthesizing 
means may employ at least one component that constitutes the 

20 frames during the estimation of the correspondent relationships, 
the obtainment of the coordinate converted frames, the 
calculation of the correlative values, the generation of the 
intermediate synthesized frames, and the generation of the 
synthesized frame . 

25 According to the first and second methods and apparatuses 

for synthesizing moving images of the present invention, a 



14 



moving image is sampled to obtain a plurality of consecutive 
frames . Next, a reference patch having at least one rectangular 
region therein, is provided in a first frame from among the 
plurality of sampled frames, which is designated as a reference 
5 frame . A second patch, which is the same as the reference patch, 
is provided in another of the plurality of sampled frames. 
Thereafter, the second patch in the other frame is moved and/or 
deformed so that an image within the second patch matches that 
within the reference image. Finally, correspondent 

10 relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 
reference frame are estimated, based on the second patch after 
the movement and/or deformation thereof and the reference 
patch . 

15 Further, a coordinate converted frame is obtained by 

coordinate converting the image within the second patch into 
a coordinate space of the reference frame, based on the 
correspondent relationships . Then, a correlative value, which 
represents the degree of correlation between the images within 

20 the reference patch of the reference frame and the coordinate 
converted frame, is calculated. 

Thereafter, the number of rectangular regions within the 
reference patch and the second patch is varied in a stepwise 
manner. The aforementioned correspondent relationships are 

25 estimated, the coordinate converted frames are obtained, and 
the correlative values are calculated for each number of 
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rectangular regions, thereby obtaining a plurality of 
correlative values corresponding to the numbers of rectangular 
regions . 

Further, the degrees of correlation are compared, for 
5 each number of rectangular regions . A synthesized frame having 
a higher resolution than either the reference frame of the other 
frame is generated, by administering interpolation 
calculations on the images within the second patch and the 
reference patch, based on the correspondent relationships, 

10 which were estimated among the pixels of the reference patch 
and the second patch having the number of rectangular regions 
therein that yielded the highest degree of correlation. 

Note that in the case that three or more frames were 
sampled, a plurality of intermediate synthesized frames are 

15 obtained, by estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
calculating the plurality of correlative values, and comparing 
the degrees of correlation, for all of the plurality of sampled 
frames. Finally a synthesized frame is generated, by 

20 synthesizing the plurality of intermediate synthesized frames. 

For this reason, a synthesized frame is generated by 
employing a number of rectangular regions, which is optimal in 
following the movement and/or deformation of a subject in the 
frames . Thereby, a synthesized frame having high image quality 

25 is enabled to be obtained, regardless of movement of the subject 
in the frames. 
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In addition, the correspondent relationships, to be 
employed in the generation of the synthesized frame or the 
intermediate synthesized frames, may be estimated for each 
rectangular region. In this case, synthesized frames or 
5 intermediate synthesized frames are generated for each 
rectangular region within the reference patch and the second 
patch. Therefore, a synthesized frame is enabled to be 
generated, employing a number of rectangular regions, which is 
optimal in following the movement of the subject in each portion 

10 of the frames. Thereby, a synthesized frame having high image 
quality is enabled to be obtained, regardless of movement of 
the subject in the frames. 

In addition, at least one component that constitutes the 
frames may be employed during the estimation of the 

15 correspondent relationships, the obtainment of the coordinate 
converted frames, the calculation of the correlative values, 
the generation of the intermediate synthesized frames (in the 
case that three or more frames are sampled) , and the generation 
of the synthesized frame. Thereby, synthesized frames can be 

20 obtained, in which deterioration of image quality is reduced 
for each component. Accordingly, a synthesized frame, 
comprising synthesized frames for each component, is enabled 
to be obtained. 

The third method for synthesizing moving images of the 

25 present invention comprises the steps of: 

sampling two consecutive frames from a moving image; 
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providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
5 reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
10 within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
15 converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 
20 patch of the reference frame and the coordinate converted frame; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
25 correlative values for each number of rectangular regions, 
until the degree of correlation is greater than or equal to a 



predetermined threshold value; and 

generating a synthesized frame having a higher resolution 
than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
5 reference patch, based on the correspondent relationships, 
which were estimated among the pixels of the reference patch 
and the second patch having the number of rectangular regions 
therein that yielded the degree of correlation greater than or 
equal to the threshold value, 

10 Here, "increasing the number of rectangular regions... 

in a stepwise manner" refers to a case in which the number of 
rectangular regions is gradually increased from a small number 
until the correlative value becomes greater than or equal to 
the predetermined threshold value. Note that if the number of 

15 rectangular regions becomes excessively large, each of the 
rectangular regions becomes too small. This will cause similar 
images to be represented in a plurality of rectangular images, 
which in turn may cause an inability for the movement and/or 
deformation of the second patch to follow the movement and/or 

20 deformation of the subject. Therefore, it is preferable that 
an upper limit is set for the number of rectangular regions. 

Here, the "correlative value' 7 may be calculated for the 
entire coordinate converted frame and the entire image within 
the reference patch of the reference frame, to represent the 

25 degree of correlation therebetween. Specifically, mean values 
or sum values of all of the correlative values for each pixel 
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within the patches may be employed. 

Here, "the degree of correlation becomes greater than or 
equal to a predetermined threshold value" refers to the 
correlative value becoming greater than or equal to the 
5 threshold value in the case that the correlative value increases 
with an increase in the degree of correlation. In the case that 
the correlative value decreases with an increase in the degree 
of correlation, the term refers to the correlative value 
becoming less than or equal to the threshold value. 
10 The fourth method for synthesizing moving images of the 

present invention comprises the steps of: 

sampling two consecutive frames from a moving image; 

providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
15 designated as a reference frame; 

providing a second patch, which is the same as the 
reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
the two frames so that an image within the second patch matches 
20 that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
25 reference patch; 

obtaining a coordinate converted frame, by coordinate 
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converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating correlative values, which represent the 
5 degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame, 
for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch, in a stepwise manner, 

10 estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
until the degree of correlation or a representative degree of 
correlation of all of the rectangular regions, or of the 

15 rectangular regions at predetermined positions from among all 
of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, or 
of a predetermined number of rectangular regions from among all 
of the rectangular regions, other than those for which the 

20 degree of correlation is less than a predetermined value, 
becomes greater than or equal to a predetermined threshold 
value; and 

generating a synthesized frame having a higher resolution 
than either of the two frames, by administering interpolation 
25 calculations on the images within the second patch and the 
reference patch for each region corresponding to the 
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rectangular regions, based on the correspondent relationships, 
which were estimated among the pixels of the reference patch 
and the second patch having the number of rectangular regions 
therein that yielded the degree of correlation or the 
5 representative degree of correlation greater than or equal to 
the threshold value. 

Here, the "correlative value" may be calculated for the 
rectangular regions of the coordinate converted frame and the 
image within the reference patch of the reference frame 

10 corresponding thereto, to represent the degree of correlation 
therebetween. Specifically, mean values or sum values of all 
of the correlative values for each pixel within the rectangular 
regions may be employed. 

Here, "those for which the degree of correlation is less 

15 than a predetermined value", from among all of the rectangular 
regions, refer to rectangular regions for which the degrees of 
correlation are extremely low, and for which the degree of 
correlation will not become greater than or equal to the 
threshold value, even if the number of rectangular regions is 

20 increased. 

The "predetermined number of rectangular regions" refers 
to at least one of "all of the rectangular regions", or at least 
one rectangular region from among "all of the rectangular 
regions, other than those for which the degree of correlation 

25 is less than a predetermined value" (hereinafter, referred to 
as "subject rectangular regions") . The "predetermined number 
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of rectangular regions" may be one, a plurality, or all of the 
subject rectangular regions. 

Note that in the case that the percentage of the 
rectangular regions within the second patch or the reference 
5 patch, for which the degree correlation is greater than or equal 
to the threshold value, is greater than or equal to a 
predetermined percentage (for example, 50%), the 
"predetermined number of rectangular regions" may be a number 
of rectangular regions for which the degree of correlation is 

10 greater than or equal to the threshold value. 

The "rectangular regions at predetermined positions" 
refers to rectangular regions within the reference patch or the 
second patch, at positions where a main subject of the frames 
is included. Specifically, rectangular regions that include 

15 a main subject such as a person's face, a complex texture, etc. 
may be employed thereas. Alternatively, rectangular regions 
positioned in the vicinity of the center of the reference patch 
or the second patch may be employed thereas. 

The "representative degree of correlation" is a value 

20 that represents the degree of correlation for the predetermined 
number of rectangular regions or for the rectangular regions 
at the predetermined positions. Specifically, a mean value, 
a median value, a maximum value, a minimum value, etc. of the 
degrees of correlation may be employed thereas. 

25 "Increasing the number of rectangular regions within the 

reference patch and the second patch, in a stepwise manner, 
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estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
until the degree of correlation or a representative degree of 
5 correlation of all of the rectangular regions, or of the 
rectangular regions at predetermined positions from among all 
of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, or 
of a predetermined number of rectangular regions from among all 

10 of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, 
becomes greater than or equal to a predetermined threshold 
value" may be performed in a variety of manners . One such manner 
is that wherein the correlative value is calculated, for the 

15 predetermined number of the subject rectangular regions or the 
subject rectangular regions at predetermined positions, for 
each number of rectangular regions, until the degree of 
correlation or the representative degree of correlation becomes 
greater than or equal to the threshold value. Alternatively, 

20 a case may be considered in which the degree of correlation or 
a representative degree of correlation for a portion of the 
predetermined number of subject rectangular regions or the 
subject rectangular regions at predetermined positions are 
greater than or equal to the threshold value. In this case, 

25 the increase in the number of rectangular regions and the 
calculation of the correlative values is only performed for the 
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rectangular regions other than the portion for which the degree 
of correlation or the representative degree of correlation is 
greater than or equal to the threshold value. In the latter 
case, the time required for calculating the correlative value 
5 can be reduced when compared to the former case. 

In the fourth method for synthesizing moving images, 
there are cases in which the number of rectangular regions that 
yields the degree of correlation or the representative degree 
of correlation differs among local areas within the reference 

10 patch and the second patch. Therefore, "administering 
interpolation calculations. . . for each region corresponding to 
the rectangular regions" refers to administering interpolation 
calculations based on different correspondent relationships, 
which have been estimated for the number of rectangular regions 

15 corresponding to different local areas within the reference 
patch and the second patch. 

Note that in the third and fourth methods for synthesizing 
moving images, the threshold value may be set to be small, 
according to the increase in the number of rectangular regions, 

20 until the number of rectangular regions reaches a predetermined 
value, and the threshold value may be set to be large, when the 
number of rectangular regions exceeds the predetermined value. 

In addition, in the third and fourth methods for 
synthesizing moving images of the present invention, at least 

25 one component that constitutes the frames may be employed during 
the estimation of the correspondent relationships, the 
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obtainment of the coordinate converted frames, the calculation 
of the correlative values, and the generation of the synthesized 
frame . 

The "at least one component that constitutes the frames'' 
5 refers to at least one of three color components RGB, in the 
case that the frames are constituted by RGB color data, for 
example. Alternatively, in the case that the frames are 
constituted by luminance chrominance components YCC, the "at 
least one component" refers to at least one of the luminance 
10 and chrominance components, and preferably the luminance 
component . 

The fifth method for synthesizing moving images of the 
present invention comprises the steps of: 

sampling at least three consecutive frames from a moving 

15 image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

20 providing a second patch, which is the same as the 

reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 

frame so that an image within the second patch matches that 

within the reference image; 
25 estimating correspondent relationships among the pixels 

within the second patch of the other frame and the pixels within 
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the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
5 converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 

10 patch of the reference frame and the coordinate converted frame; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 

15 correlative values for each number of rectangular regions, 
until the degree of correlation is greater than or equal to a 
predetermined threshold value; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 

20 frame, by administering interpolation calculations on the 
images within the second patch and the reference patch for each 
region corresponding to the rectangular regions, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 

25 number of rectangular regions therein that yielded the degree 
of correlation greater than or equal to the threshold value; 
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obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
plurality of correlative values, for all of the plurality of 
5 sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

The sixth method for synthesizing moving images of the 
present invention comprises the steps of: 
10 sampling at least three consecutive frames from a moving 

image ; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
15 frame; 

providing a second patch, which is the same as the 
reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
20 within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
25 reference patch; 

obtaining a coordinate converted frame, by coordinate 
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converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a correlative value, which represents the 
5 degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 

10 obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
until the degree of correlation or a representative degree of 
correlation of all of the rectangular regions, or of the 
rectangular regions at predetermined positions from among all 

15 of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, or 
of a predetermined number of rectangular regions from among all 
of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, 

20 becomes greater than or equal to a predetermined threshold 
value; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 
25 images within the second patch and the reference patch for each 
region corresponding to the rectangular regions, based on the 
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correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the degree 
of correlation or the representative degree of correlation 
5 greater than or equal to the threshold value; 

obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
plurality of correlative values, for all of the plurality of 

10 sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

Note that in the fifth and sixth methods for synthesizing 
moving images, the threshold value may be set to be small, 

15 according to the increase in the number of rectangular regions, 
until the number of rectangular regions reaches a predetermined 
value, and the threshold value may be set to be large, when the 
number of rectangular regions exceeds the predetermined value. 

In addition, in the fifth and sixth methods for 

20 synthesizing moving images of the present invention, at least 
one component that constitutes the frames maybe employed during 
the estimation of the correspondent relationships, the 
obtainment of the coordinate converted frames, the calculation 
of the correlative values, the obtainment of the intermediate 

25 synthesized frames, and the generation of the synthesized 
frame . 
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The third moving image synthesizing apparatus of the 
present invention comprises: 

a sampling means, for sampling two consecutive frames 
from a moving image; 
5 a correspondent relationship estimating means, for: 

providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
same as the reference patch, in the other of the two frames; 

10 moving and/or deforming the second patch in the other of the 
two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 

15 reference frame, based on the second patch after the movement 
and/or deformation thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 

20 based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
frame and the coordinate converted frame; 

25 a control means, for controlling the correspondent 

relationship estimating means, the coordinate converting means, 
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and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner, to estimate the 
aforementioned correspondent relationships, to obtain the 
5 coordinate converted frames, and to calculate the correlative 
values for each number of rectangular regions, until the degree 
of correlation is greater than or equal to a predetermined 
threshold value; and 

a synthesizing means, for generating a synthesized frame 

10 having a higher resolution than either of the two frames, by 
administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 

15 number of rectangular regions therein that yielded the degree 
of correlation greater than or equal to the predetermined 
threshold value. 

The fourth moving image synthesizing apparatus of the 
present invention comprises: 

20 a sampling means, for sampling two consecutive frames 

from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 

25 as a reference frame; providing a second patch, which is the 
same as the reference patch, in the other of the two frames; 
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moving and/or deforming the second patch in the other of the 
two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
5 other frame and the pixels within the reference patch of the 
reference frame, based on the second patch after the movement 
and/or deformation thereof and the reference patch; 

a coordinate converting means , for obtaining a coordinate 
converted frame, by coordinate converting the image within the 

10 second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 

15 frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 

20 the second patch in a stepwise manner, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
values for each number of rectangular regions, until the degree 
of correlation or a representative degree of correlation of all 

25 of the rectangular regions, or of the rectangular regions at 
predetermined positions from among all of the rectangular 



33 



regions, other than those for which the degree of correlation 
is less than a predetermined value, or of a predetermined number 
of rectangular regions from among all of the rectangular regions, 
other than those for which the degree of correlation is less 
5 than a predetermined value, becomes greater than or equal to 
a predetermined threshold value; and 

a synthesizing means, for generating a synthesized frame 
having a higher resolution than either of the two frames, by 
administering interpolation calculations on the images within 

10 the second patch and the. reference patch for each region 
corresponding to the rectangular regions, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the degree 

15 of correlation or the representative degree of correlation 
greater than or equal to the predetermined threshold value. 

Note that in the third and fourth moving image 
synthesizing apparatuses, a construction may be adopted 
wherein: 

20 the control means decreases the threshold value, 

according to the increase in the number of rectangular regions, 
until the number of rectangular regions reaches a predetermined 
value, and increases the threshold value, when the number of 
rectangular regions exceeds the predetermined value. 

25 In addition, in the third and fourth moving image 

synthesizing apparatus, a construction may be adopted wherein: 
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the correspondent relationship estimating means, the 
coordinate converting means, the correlative value calculating 
means, and the synthesizing means employ at least one component 
that constitutes the frames during the estimation of the 
5 correspondent relationships, the obtainment of the coordinate 
converted frames, the calculation of the correlative values, 
and the generation of the synthesized frame. 

The fifth moving image synthesizing apparatus of the 
present invention comprises: 

10 a sampling means, for sampling at least three consecutive 

frames from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 
region therein, in a first frame from among the at least three 

15 sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 
patch, in another of the plurality of sampled frames; moving 
and/or deforming the second patch in the other frame so that 
an image within the second patch matches that within the 

20 reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 
the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 
thereof and the reference patch; 

25 a coordinate converting means, for obtaining a coordinate 

converted frame, by coordinate converting the image within the 
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second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
values for each number of rectangular regions, until the degree 
of correlation is greater than or equal to a predetermined 
threshold value; and 

a synthesizing means, for: obtaining an intermediate 
synthesized frame having a higher resolution than either the 
reference frame of the other frame, by: administering 
interpolation calculations on the images within the second 
patch and the reference patch, based on the correspondent 
relationships, which were estimated among the pixels of the 
reference patch and the second patch having the number of 
rectangular regions therein that yielded the degree of 
correlation greater than or equal to the threshold value; 
obtaining a plurality of intermediate synthesized frames by 
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estimating the aforementioned correspondent relationships 
obtaining the coordinate converted frames, and calculating the 
plurality of correlative values, for all of the plurality of 
sampled frames; and generating a synthesized frame by 
5 synthesizing the plurality of intermediate synthesized frames . 

The sixth moving image synthesizing apparatus of the 
present invention comprises: 

a sampling means, for sampling at least three consecutive 
frames from a moving image; 

10 a correspondent relationship estimating means, for: 

providing a reference patch having at least one rectangular 
region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 

15 patch, in another of the plurality of sampled frames; moving 
and/or deforming the second patch in the other frame so that 
an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 

20 the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 
thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 

25 second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 
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a correlative value calculating means, for calculating 
a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
frame and the coordinate converted frame; 
5 a control means, for controlling the correspondent 

relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner, to estimate the 

10 aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
values for each number of rectangular regions, until the degree 
of correlation or a representative degree of correlation of all 
of the rectangular regions, or of the rectangular regions at 

15 predetermined positions from among all of the rectangular 
regions, other than those for which the degree of correlation 
is less than a predetermined value, or of a predetermined number 
of rectangular regions from among all of the rectangular regions, 
other than those for which the degree of correlation is less 

20 than a predetermined value, becomes greater than or equal to 
a predetermined threshold value; and 

a synthesizing means, for: obtaining an intermediate 
synthesized frame having a higher resolution than either the 
reference frame of the other frame, by: administering 

25 interpolation calculations on the images within the second 
patch and the reference patch for each region corresponding to 



the rectangular regions, based on the correspondent 
relationships, which were estimated among the pixels of the 
reference patch and the second patch having the number of 
rectangular regions therein that yielded the degree of 
5 correlation or the representative degree of correlation greater 
than or equal to the threshold value; obtaining a plurality of 
intermediate synthesized frames by estimating the 
aforementioned correspondent relationships obtaining the 
coordinate converted frames, and calculating the plurality of 

10 correlative values, for all of the plurality of sampled frames; 
and generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

Note that in the fifth and sixth moving image synthesizing 
apparatuses, a construction may be adopted wherein: 

15 the control means decreases the threshold value, 

according to the increase in the number of rectangular regions, 
until the number of rectangular regions reaches a predetermined 
value, and increases the threshold value, when the number of 
rectangular regions exceeds the predetermined value. 

20 In addition, in the fifth and sixth moving image 

synthesizing apparatus, a construction may be adopted wherein: 
the correspondent relationship estimating means, the 
coordinate converting means, the correlative value calculating 
means, and the synthesizing means employ at least one component 

25 that constitutes the frames during the estimation of the 
correspondent relationships, the obtainment of the coordinate 
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converted frames, the calculation of the correlative values, 
the obtainment of the intermediate synthesized frames, and the 
generation of the synthesized frame. 

According to the third method and the third apparatus for 
5 synthesizing moving images of the present invention, a moving 
image is sampled to obtain a plurality of consecutive frames. 
Next, a reference patch having at least one rectangular region 
therein, is provided in a first frame from among the plurality 
of sampled frames, which is designated as a reference frame. 

10 A second patch, which is the same as the reference patch, is 
provided in another of the plurality of sampled frames. 
Thereafter, the second patch in the other frame is moved and/or 
deformed so that an image within the second patch matches that 
within the reference image. Finally, correspondent 

15 relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 
reference frame are estimated, based on the second patch after 
the movement and/or deformation thereof and the reference 
patch. 

20 Further, a coordinate converted frame is obtained by 

coordinate converting the image within the second patch into 
a coordinate space of the reference frame, based on the 
correspondent relationships . Then, a correlative value, which 
represents the degree of correlation between the images within 

25 the reference patch of the reference frame and the coordinate 
converted frame, is calculated. Thereafter, the number of 



rectangular regions within the reference patch and the second 
patch is increased in a stepwise manner. The aforementioned 
correspondent relationships are estimated, the coordinate 
converted frames are obtained, and the correlative values are 
5 calculated for each number of rectangular regions. The number 
of rectangular regions within the reference patch and the second 
patch are increased until the degree of correlation becomes 
greater than or equal to the threshold value. 

A synthesized frame having a higher resolution than 

10 either the reference frame of the other frame is generated, by 
administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 

15 number of rectangular regions therein that yielded the degree 
of correlation greater than or equal to the threshold value. 

Note that in the case that three or more frames are sampled, 
as in the fifth method and the fifth apparatus for synthesizing 
moving images, a plurality of intermediate synthesized frames 

20 are obtained, by estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the plurality of correlative values, for all of the 
plurality of sampled frames. Finally a synthesized frame is 
generated, by synthesizing the plurality of intermediate 

25 synthesized frames. 

For this reason, a synthesized frame is generated by 

41 



employing a number of rectangular regions, which is optimal in 
following the movement and/or deformation of a subject in the 
frames. Thereby, a synthesized frame having high image quality 
is enabled to be obtained, regardless of movement of the subject 
5 in the frames* 

According to the fourth method and the fourth apparatus 
for synthesizing moving images of the present invention, a 
moving image is sampled to obtain a plurality of consecutive 
frames . Next, a reference patch having at least one rectangular 

10 region therein, is provided in a first frame from among the 
plurality of sampled frames, which is designated as a reference 
frame . A second patch, which is the same as the reference patch, 
is provided in another of the plurality of sampled frames. 
Thereafter, the second patch in the other frame is moved and/or 

15 deformed so that an image within the second patch matches that 
within the reference image. Finally, correspondent 

relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 
reference frame are estimated, based on the second patch after 

2 0 the movement and/or deformation thereof and the reference 
patch. 

Further, a coordinate converted frame is obtained by 
coordinate converting the image within the second patch into 
a coordinate space of the reference frame, based on the 
25 correspondent relationships . Then, a correlative value, which 
represents the degree of correlation between the images within 



42 



the reference patch of the reference frame and the coordinate 
converted frame, is calculated. Thereafter, the number of 
rectangular regions within the reference patch and the second 
patch is increased in a stepwise manner. The aforementioned 
5 correspondent relationships are estimated, the coordinate 
converted frames are obtained, and the correlative values are 
calculated for each number of rectangular regions. The number 
of rectangular regions within the reference patch and the second 
patch are increased until the degree of correlation or the 

10 representative degree of correlation of all of the rectangular 
regions, or of the rectangular regions at predetermined 
positions from among all of the rectangular regions, other than 
those for which the degree of correlation is less than a 
predetermined value, or of a predetermined number of 

15 rectangular regions from among all of the rectangular regions, 
other than those for which the degree of correlation is less 
than a predetermined value, becomes greater than or equal to 
a predetermined threshold value. 

A synthesized frame having a higher resolution than 

20 either the reference frame of the other frame is generated, by 
administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 

25 number of rectangular regions therein that yielded the degree 
of correlation or the representative degree of correlation 
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greater than or equal to the threshold value. 

Note that in the case that three or more frames are sampled, 
as in the sixth method and the sixth apparatus for synthesizing 
moving images, a plurality of intermediate synthesized frames 
5 are obtained, by estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the plurality of correlative values, for all of the 
plurality of sampled frames. Finally a synthesized frame is 
generated, by synthesizing the plurality of intermediate 

10 synthesized frames. 

For this reason, a synthesized frame is generated by 
employing a number of rectangular regions, which is optimal in 
following the movement and/or deformation of a subject in the 
frames . Thereby, a synthesized frame having high image quality 

15 is enabled to be obtained, regardless of movement of the subject 
in the frames . 

Note that the threshold value, to be compared against the 
degree of correlation or the representative degree of 
correlation, may be set to be small, according to the increase 

20 in the number of rectangular regions, until the number of 
rectangular regions reaches a predetermined value, and the 
threshold value may be set to be large, when the number of 
rectangular regions exceeds the predetermined value. 

In addition, at least one component that constitutes the 

25 frames may be employed during the estimation of the 
correspondent relationships, the obtainment of the coordinate 
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converted frames, the calculation of the correlative values, 
the generation of the intermediate synthesized frames (in the 
case that three or more frames are sampled) , and the generation 
of the synthesized frame. Thereby, synthesized frames can be 
5 obtained, in which deterioration of image quality is reduced 
for each component. Accordingly, a synthesized frame, 
comprising synthesized frames for each component, is enabled 
to be obtained. 

The seventh method for synthesizing moving images of the 
10 present invention comprises the steps of: 

sampling two consecutive frames from a moving image; 
providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 
15 providing a second patch, which is the same as the 

reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
the two frames so that an image within the second patch matches 
that within the reference image; 
20 estimating correspondent relationships among the pixels 

within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

25 obtaining a coordinate converted frame, by coordinate 

converting the image within the second patch into a coordinate 
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space of the reference frame, based on the correspondent 

relationships; 

calculating a first correlative value, which represents 

the degree of correlation between the images within the 
5 reference patch of the reference frame and the coordinate 

converted frame; 

increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner by 

one step, estimating the aforementioned correspondent 
10 relationships, obtaining the coordinate converted frames, and 

calculating the correlative value, thereby obtaining a second 

correlative value; 

calculating the variation between the first and second 

correlative values ; 
15 increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner an 

additional step, estimating the aforementioned correspondent 

relationships, obtaining the coordinate converted frames, 

calculating the first and second correlative values, and 
20 calculating the variation between the first and second 

correlative values of consecutive steps, until the variation 

becomes less than or equal to a predetermined threshold value; 

and 

generating a synthesized frame having a higher resolution 
25 than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
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reference patch, based on the correspondent relationships, 
which were estimated among the pixels of either of the reference 
patches and the second patches having the number of rectangular 
regions therein that yielded the variation less than or equal 
5 to the threshold value. 

Here, "increasing the number of rectangular regions... 
in a stepwise manner an additional step" refers to increasing 
the number of rectangular regions one step in the stepwise 
manner . 

10 Here, the "variation between the first and second 

correlative values" refers to a value that represents the amount 
by which the first and second correlative values differ. The 
first correlative value includes that which was calculated for 
the initial step in which the correspondent relationship was 

15 first estimated. The second correlative value is that which 
was calculated for the number of rectangular regions one step 
greater than that for which the first correlative value was 
calculated. Specifically, the difference between the two 
correlative values, or an absolute value of the difference may 

20 be employed as the variation. 

The eighth method for synthesizing moving images of the 
present invention comprises the steps of: 

sampling two consecutive frames from a moving image; 
providing a reference patch having at least one 

25 rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 
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providing a second patch, which is the same as the 
reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
the two frames so that an image within the second patch matches 
5 that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
10 reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

15 calculating a first correlative value, which represents 

the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
converted frame, for each of the rectangular regions; 

increasing the number of rectangular regions within the 

20 reference patch and the second patch in a stepwise manner by 
one step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
correlative value; 

25 calculating the variation between the first and second 

correlative values for each of the rectangular regions; 



increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
5 calculating the first and second correlative values, and 
calculating the variation between the first and second 
correlative values of consecutive steps, until the variation 
or a representative variation of all of the rectangular regions, 
or of the rectangular regions at predetermined positions from 

10 among all of the rectangular regions, other than those for which 
the variation is greater than or equal to a predetermined value, 
or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 

15 becomes less than or equal to a predetermined threshold value; 
and 

generating a synthesized frame having a higher resolution 
than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 

20 reference patch for each region corresponding to the 
rectangular regions, based on the correspondent relationships, 
which were estimated among the pixels of either of the reference 
patches and the second patches having the number of rectangular 
regions therein that yielded the variation or the 

25 representative variation less than or equal to the threshold 
value. 
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Here, "those for which the variation is greater than or 
equal to a predetermined value", from among all of the 
rectangular regions, refer to rectangular regions for which the 
degrees of correlation are extremely low, and for which the 
5 variation will not become less than or equal to the threshold 
value, even if the number of rectangular regions is increased. 

The "predetermined number of rectangular regions" refers 
to at least one of "all of the rectangular regions", or at least 
one rectangular region from among "all of the rectangular 

10 regions, other than those for which the degree of correlation 
is less than a predetermined value" (hereinafter, referred to 
as "subject rectangular regions") . The "predetermined number 
of rectangular regions" may be one, a plurality, or all of the 
subject rectangular regions. 

15 Note that in the case that the percentage of the 

rectangular regions within the second patch or the reference 
patch, for which the variation is less than or equal to the 
threshold value, is greater than or equal to a predetermined 
percentage (for example, 50%), the "predetermined number of 

20 rectangular regions" may be a number of rectangular regions for 
which the variation is less than or equal to the threshold value . 

The "rectangular regions at predetermined positions" 
refers to rectangular regions within the reference patch or the 
second patch, at positions where a main subject of the frames 

25 is included. Specifically, rectangular regions that include 
a main subject such as a person's face, a complex texture, etc. 
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may be employed thereas. Alternatively, rectangular regions 
positioned in the vicinity of the center of the reference patch 
or the second patch may be employed thereas. 

The "representative variation" is a value that represents 
5 the variation for the predetermined number of rectangular 
regions or for the rectangular regions at the predetermined 
positions. Specifically, a mean value, a median value, a 
maximum value, a minimum value, etc. of the variation may be 
employed thereas. 

10 "Increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 

15 calculating the variation between the first and second 
correlative values of consecutive steps, until the variation 
or a representative variation of all of the rectangular regions, 
or of the rectangular regions at predetermined positions from 
among all of the rectangular regions, other than those for which 

20 the variation is greater than or equal to a predetermined value, 
or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value' 7 

25 may be performed in a variety of manners. One such manner is 
that wherein the variation is calculated, for the predetermined 
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number of the subject rectangular regions or the subject 
rectangular regions at predetermined positions, for each number 
of rectangular regions, until the variation or the 
representative variation becomes less than or equal to the 
5 threshold value. Alternatively, a case may be considered in 
which the variation or a representative variation for a portion 
of the predetermined number of subject rectangular regions or 
the subject rectangular regions at predetermined positions are 
less than or equal to the threshold value. In this case, the 

10 increase in the number of rectangular regions and the 
calculation of the variation is only performed for the 
rectangular regions other than the portion for which the 
variation or the representative variation is less than or equal 
to the threshold value. In the latter case, the time required 

15 for calculating the variation can be reduced when compared to 
the former case. 

In the eighth method for synthesizing moving images, 
there are cases in which the number of rectangular regions that 
yields the variation or the representative variation differs 

20 among local areas within the reference patch and the second 
patch. Therefore, "administering interpolation 

calculations... for each region corresponding to the 
rectangular regions" refers to administering interpolation 
calculations based on different correspondent relationships, 

25 which have been estimated for the number of rectangular regions 
corresponding to different local areas within the reference 
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patch and the second patch. 

Note that in the seventh and eighth methods for 
synthesizing moving images of the present invention, the 
predetermined threshold value may be set to be larger, according 
5 to the increase in the number of rectangular regions, when the 
number of rectangular regions exceeds a predetermined value. 

In addition, in the seventh and eighth methods for 
synthesizing moving images, at least one component that 
constitutes the frames may be employed during the estimation 

10 of the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
values, the calculation of the variations, and the generation 
of the synthesized frame. 

The "at least one component that constitutes the frames" 

15 refers to at least one of three color components RGB, in the 
case that the frames are constituted by RGB color data, for 
example. Alternatively, in the case that the frames are 
constituted by luminance chrominance components YCC, the "at 
least one component"' refers to at least one of the luminance 

2 0 and chrominance components, and preferably the luminance 
component . 

The ninth method for synthesizing moving images of the 
present invention comprises the steps of: 

sampling at least three consecutive frames from a moving 

25 image; 

providing a reference patch having at least one 
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rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

providing a second patch, which is the same as the 
5 reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
within the reference image; 

estimating correspondent relationships among the pixels 
10 within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
15 converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a first correlative value, which represents 
the degree of correlation between the images within the 
20 reference patch of the reference frame and the coordinate 
converted frame; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner by 
one step, estimating the aforementioned correspondent 
25 relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
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correlative valuer- 
calculating the variation between the first and second 
correlative values ; 

increasing the number of rectangular regions within the 
5 reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 
calculating the variation between the first and second 

10 correlative values of consecutive steps, until the variation 
becomes less than or equal to a predetermined threshold value; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 

15 images within the second patch and the reference patch, based 
on the correspondent relationships, which were estimated among 
the pixels of the reference patch and the second patch having 
the number of rectangular regions therein that yielded the 
variation less than or equal to the threshold value; 

20 obtaining a plurality of intermediate synthesized frames 

by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and calculating the variations 
among the plurality of the correlative values, for all of the 

25 plurality of sampled frames; and 

generating a synthesized frame by synthesizing the 
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plurality of intermediate synthesized frames. 

The tenth method for synthesizing moving images of the 
present invention comprises the steps of: 

sampling at least three consecutive frames from a moving 

5 image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

10 providing a second patch, which is the same as the 

reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
within the reference image; 

15 estimating correspondent relationships among the pixels 

within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

20 obtaining a coordinate converted frame, by coordinate 

converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a first correlative value, which represents 

25 the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
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converted frame, for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner by 
one step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
correlative value ; 

calculating the variation between the first and second 
correlative values for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 
calculating the variation between the first and second 
correlative values of consecutive steps, until the variation 
or a representative variation of all of the rectangular regions, 
or of the rectangular regions at predetermined positions from 
among all of the rectangular regions, other than those for which 
the variation is greater than or equal to a predetermined value, 
or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
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frame, by administering interpolation calculations on the 
images within the second patch and the reference patch for each 
region corresponding to the rectangular regions, based on the 
correspondent relationships, which were estimated among the 
5 pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the 
variation or the representative variation less than or equal 
to the threshold value; 

obtaining a plurality of intermediate synthesized frames 

10 by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and calculating the variations 
among the plurality of the correlative values, for all of the 
plurality of sampled frames; and 

15 generating a synthesized frame by synthesizing the 

plurality of intermediate synthesized frames. 

Note that in the ninth and tenth methods for synthesizing 
moving images of the present invention, the predetermined 
threshold value may be set to be larger, according to the 

20 increase in the number of rectangular regions, when the number 
of rectangular regions exceeds a predetermined value. 

In addition, in the ninth and tenth methods for 
synthesizing moving images, at least one component that 
constitutes the frames may be employed during the estimation 

25 of the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
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values, the calculation of the variations, the obtainment of 
the intermediate synthesized frames, and the generation of the 
synthesized frame . 

The seventh moving image synthesizing apparatus of the 
5 present invention comprises: 

a sampling means, for sampling two consecutive frames 
from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 

10 region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
same as the reference patch, in the other of the two frames; 
moving and/or deforming the second patch in the other of the 
two frames so that an image within the second patch matches that 

15 within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 
reference frame, based on the second patch after the movement 
and/or deformation thereof and the reference patch; 

20 a coordinate converting means, for obtaining a coordinate 

converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 

25 a first correlative value, which represents the degree of 
correlation between the images within the reference patch of 
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the reference frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
5 number of rectangular regions within the reference patch and 
the second patch in a stepwise manner by one step, to estimate 
the aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
value, thereby obtaining a second correlative value; 

10 a variation calculating means, for calculating the 

variation between the first and second correlative values; 

a comparing means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
the correlative relationship calculating means, and the 

15 variation calculating means, to increase the number of 
rectangular regions within the reference patch and the second 
patch in a stepwise manner an additional step, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, to calculate the first and second 

20 correlative values, and to calculate the variation between the 
first and second correlative values of consecutive steps, until 
the variation becomes less than or equal to a predetermined 
threshold value; and 

a synthesizing means, for generating a synthesized frame 

25 having a higher resolution than either of the two frames, by 
administering interpolation calculations on the images within 
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the second patch and the reference patch, based on the 
correspondent relationships, which were estimated among the 
pixels of either of the reference patches and the second patches 
having the number of rectangular regions therein that yielded 
5 the variation less than or equal to the threshold value. 

The eighth moving image synthesizing apparatus of the 
present invention comprises: 

a sampling means, for sampling two consecutive frames 
from a moving image; 

10 a correspondent relationship estimating means, for: 

providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
same as the reference patch, in the other of the two frames; 

15 moving and/or deforming the second patch in the other of the 
two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 

20 reference frame, based on the second patch after the movement 
and/or deformation thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 

25 based on the correspondent relationships; 

a correlative value calculating means, for calculating 
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a first correlative value, which represents the degree of 
correlation between the images within the reference patch of 
the reference frame and the coordinate converted frame, for each 
of the rectangular regions; 
5 a control means, for controlling the correspondent 

relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner by one step, to estimate 

10 the aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
value, thereby obtaining a second correlative value; 

a variation calculating means, for calculating the 
variation between the first and second correlative, values for 

15 each of the rectangular regions; 

a comparing means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
the correlative relationship calculating means, and the 
variation calculating means, to increase the number of 

20 rectangular regions within the reference patch and the second 
patch in a stepwise manner an additional step, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, to calculate the first and second 
correlative values, and to calculate the variation between the 

25 first and second correlative values of consecutive steps, until 
the variation or a representative variation of all of the 
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rectangular regions, or of the rectangular regions at 
predetermined positions from among all of the rectangular 
regions, other than those for which the variation is greater 
than or equal to a predetermined value, or of a predetermined 
5 number of rectangular regions from among all of the rectangular 
regions, other than those for which the variation is greater 
than or equal to a predetermined value, becomes less than or 
equal to a predetermined threshold value; and 

a synthesizing means, for generating a synthesized frame 

10 having a higher resolution than either of the two frames, by 
administering interpolation calculations on the images within 
the second patch and the reference patch for each region 
corresponding to the rectangular regions, based on the 
correspondent relationships, which were estimated among the 

15 pixels of either of the reference patches and the second patches 
having the number of rectangular regions therein that yielded 
the variation or the representative variation less than or equal 
to the threshold value. 

Note that in the seventh and eighth moving image 

20 synthesizing apparatuses of the present invention, the control 
means may set the predetermined threshold value to be larger, 
according to the increase in the number of rectangular regions, 
when the number of rectangular regions exceeds a predetermined 
value . 

25 In addition, in the seventh and eighth moving image 

synthesizing apparatuses of the present invention, the 



correspondent relationship estimating means, the coordinate 
converting means, the correlative value calculating means, the 
control means, the variation calculating means, the comparing 
means, and the synthesizing means may employ at least one 
component that constitutes the frames during the estimation of 
the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
values, the calculation of the variations, and the generation 
of the synthesized frame. 

The ninth moving image synthesizing apparatus of the 
present invention comprises: 

a sampling means, for sampling at least three consecutive 
frames from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 
region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 
patch, in another of the plurality of sampled frames; moving 
and/or deforming the second patch in the other frame so that 
an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 
the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 
thereof and the reference patch; 



64 



a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 
5 a correlative value calculating means, for calculating 

a first correlative value, which represents the degree of 
correlation between the images within the reference patch of 
the reference frame and the coordinate converted frame; 

a control means, for controlling the correspondent 

10 relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner by one step, to estimate 
the aforementioned correspondent relationships, to obtain the 

15 coordinate converted frames, and to calculate the correlative 
value, thereby obtaining a second correlative value; 

a variation calculating means, for calculating the 
variation between the first and second correlative values; 

a comparing means, for controlling the correspondent 

20 relationship estimating means, the coordinate converting means , 
the correlative value calculating means, and the variation 
calculating means, to increase the number of rectangular 
regions within the reference patch and the second patch in a 
stepwise manner an additional step, to estimate the 

25 aforementioned correspondent relationships, to obtain the 
coordinate converted frames, to calculate the first and second 
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correlative values, and to calculate the variation between the 
first and second correlative values of consecutive steps, until 
the variation becomes less than or equal to a predetermined 
threshold value; and 

a synthesizing means, for: obtaining an intermediate 
synthesized frame having a higher resolution than either the 
reference frame of the other frame, by administering 
interpolation calculations on the images within the second 
patch and the reference patch, based on the correspondent 
relationships, which were estimated among the pixels of the 
reference patch and the second patch having the number of 
rectangular regions therein that yielded the variation less 
than or equal to the threshold value; obtaining a plurality of 
intermediate synthesized frames by estimating the 
aforementioned correspondent relationships, obtaining the 
coordinate converted frames, calculating the plurality of 
correlative values, and calculating the variations among the 
plurality of the correlative values, for all of the plurality 
of sampled frames; and generating a synthesized frame by 
synthesizing the plurality of intermediate synthesized frames. 

The tenth moving image synthesizing apparatus of the 
present invention comprises: 

a sampling means, for sampling at least three consecutive 
frames from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 



region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 
patch, in another of the plurality of sampled frames; moving 
5 and/or deforming the second patch in the other frame so that 
an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 
the pixels within the reference patch of the reference frame, 

10 based on the second patch after the movement and/or deformation 
thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 

15 based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a first correlative value, which represents the degree of 
correlation between the images within the reference patch of 
the reference frame and the coordinate converted frame, for each 

20 of the rectangular regions ; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 

25 the second patch in a stepwise manner by one step, to estimate 
the aforementioned correspondent relationships, to obtain the 



coordinate converted frames, and to calculate the correlative 
value, thereby obtaining a second correlative value; 

a variation calculating means, for calculating the 
variation between the first and second correlative values for 
5 each of the rectangular regions; 

a comparing means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
the correlative value calculating means, and the variation 
calculating means, to increase the number of rectangular 

10 regions within the reference patch and the second patch in a 
stepwise manner an additional step, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, to calculate the first and second 
correlative values, and to calculate the variation between the 

15 first and second correlative values of consecutive steps, until 
the variation of all of the rectangular regions, or of the 
rectangular regions at predetermined positions from among all 
of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 

20 or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value; 
and 

25 a synthesizing means, for: obtaining an intermediate 

synthesized frame having a higher resolution than either the 
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reference frame of the other frame, by administering 
interpolation calculations on the images within the second 
patch and the reference patch, based on the correspondent 
relationships, which were estimated among the pixels of the 
5 reference patch and the second patch having the number of 
rectangular regions therein that yielded the variation or the 
representative variation less than or equal to the threshold 
value; obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 

10 obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and calculating the variations 
among the plurality of the correlative values, for all of the 
plurality of sampled frames; and generating a synthesized frame 
by synthesizing the plurality of intermediate synthesized 

15 frames. 

Note that in the ninth and tenth moving image synthesizing 
apparatuses of the present invention, the control means may set 
the predetermined threshold value to be larger, according to 
the increase in the number of rectangular regions, when the 

20 number of rectangular regions exceeds a predetermined value. 

In addition, in the ninth and tenth moving image 
synthesizing apparatuses of the present invention, the 
correspondent relationship estimating means, the coordinate 
converting means, the correlative value calculating means, the 

25 control means, the variation calculating means, the comparing 
means, and the synthesizing means may employ at least one 
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component that constitutes the frames during the estimation of 
the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
values, the calculation of the variations, the obtainment of 
5 the intermediate synthesized frames, and the generation of the 
synthesized frame. 

According to the seventh method and the seventh apparatus 
for synthesizing moving images of the present invention, a 
moving image is sampled to obtain a plurality of consecutive 

10 frames. Next, a reference patch having at least one rectangular 
region therein, is provided in a first frame from among the 
plurality of sampled frames, which is designated as a reference 
frame . A second patch, which is the same as the reference patch, 
is provided in another of the plurality of sampled frames. 

15 Thereafter, the second patch in the other frame is moved and/or 
deformed so that an image within the second patch matches that 
within the reference image. Finally, correspondent 

relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 

20 reference frame are estimated, based on the second patch after 
the movement and/or deformation thereof and the reference 
patch. 

Further, a coordinate converted frame is obtained by 
coordinate converting the image within the second patch into 
25 a coordinate space of the reference frame, based on the 
correspondent relationships. Then, the first correlative 
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value, which represents the degree of correlation between the 
images within the reference patch of the reference frame and 
the coordinate converted frame, is calculated. Thereafter, 
the number of rectangular regions within the reference patch 
5 and the second patch is increased in a stepwise manner by an 
additional step. The aforementioned correspondent 

relationships are estimated, the coordinate converted frames 
are obtained, and the second correlative value is calculated. 
Then, the variation between the first and second correlative 

10 values is calculated. 

Thereafter, the number of rectangular regions within the 
reference patch and the second patch are increased in a stepwise 
manner, the aforementioned correspondent relationships are 
estimated, the coordinate converted frames are obtained, the 

15 first and second correlative values are calculated, and the 
variation between the first and second correlative values of 
consecutive steps are calculated, until the variation becomes 
less than or equal to a predetermined threshold value. 

A synthesized frame having a higher resolution than 

20 either the reference frame of the other frame is generated, by 
administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 

25 number of rectangular regions therein that yielded the 
variation less than or equal to the threshold value. 
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Note that in the case that three or more frames are sampled, 
as in the ninth method and the ninth apparatus for synthesizing 
moving images, a plurality of intermediate synthesized frames 
are obtained, by estimating the aforementioned correspondent 
5 relationships, obtaining the coordinate converted frames, 
calculating the plurality of correlative values, and 
calculating the variations for all of the plurality of sampled 
frames. Finally a synthesized frame is generated, by 
synthesizing the plurality of intermediate synthesized frames . 

10 For this reason, a synthesized frame is generated by 

employing a number of rectangular regions, which is optimal in 
following the movement and/or deformation of a subject in the 
frames . Thereby, a synthesized frame having high image quality 
is enabled to be obtained, regardless of movement of the subject 

15 in the frames . 

According to the eighth method and the eighth apparatus 
for synthesizing moving images of the present invention, a 
moving image is sampled to obtain a plurality of consecutive 
frames . Next, a reference patch having at least one rectangular 

20 region therein, is provided in a first frame from among the 
plurality of sampled frames, which is designated as a reference 
frame . A second patch, which is the same as the reference patch, 
is provided in another of the plurality of sampled frames . 
Thereafter, the second patch in the other frame is moved and/or 

25 deformed so that an image within the second patch matches that 
within the reference image. Finally, correspondent 
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relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 
reference frame are estimated, based on the second patch after 
the movement and/or deformation thereof and the reference 
5 patch. 

Further, a coordinate converted frame is obtained by 
coordinate converting the image within the second patch into 
a coordinate space of the reference frame, based on the 
correspondent relationships. Then, the first correlative 

10 value, which represents the degree of correlation between the 
images within the reference patch of the reference frame and 
the coordinate converted frame, is calculated. Thereafter, 
the number of rectangular regions within the reference patch 
and the second patch is increased in a stepwise manner by an 

15 additional step. The aforementioned correspondent 

relationships are estimated, the coordinate converted frames 
are obtained, and the second correlative value is calculated. 
Then, the variation between the first and second correlative 
values is calculated. 

20 Thereafter, the number of rectangular regions within the 

reference patch and the second patch are increased in a stepwise 
manner, the aforementioned correspondent relationships are 
estimated, the coordinate converted frames are obtained, the 
first and second correlative values are calculated, and the 

25 variation between the first and second correlative values of 
consecutive steps are calculated, until the variation or the 
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representative variation of all of the rectangular regions, or 
of the rectangular regions at predetermined positions from 
among all of the rectangular regions, other than those for which 
the variation is greater than or equal to a predetermined value, 
5 or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value. 

A synthesized frame having a higher resolution than 

10 either the reference frame of the other frame is generated, by 
administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 

15 number of rectangular regions therein that yielded the 
variation or the representative variation less than or equal 
to the threshold value. 

Note that in the case that three or more frames are sampled, 
as in the tenth method and the tenth apparatus for synthesizing 

20 moving images, a plurality of intermediate synthesized frames 
are obtained, by estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
calculating the plurality of correlative values, and 
calculating the variations for all of the plurality of sampled 

25 frames. Finally a synthesized frame is generated, by 
synthesizing the plurality of intermediate synthesized frames. 
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For this reason, a synthesized frame is generated by 
employing a number of rectangular regions, which is optimal in 
following the movement and/or deformation of a subject in the 
frames. Thereby, a synthesized frame having high image quality 
5 is enabled to be obtained, regardless of movement of the subject 
in the frames. 

Note that the threshold value, to be compared against the 
variation or the representative variation, may be set to be 
large, according to the increase in the number of rectangular 

10 regions, when the number of rectangular regions exceeds a 
predetermined value. 

In addition, at least one component that constitutes the 
frames may be employed during the estimation of the 
correspondent relationships, the obtainment of the coordinate 

15 converted frames, the calculation of the correlative values, 
the generation of the intermediate synthesized frames (in the 
case that three or more frames are sampled) , and the generation 
of the synthesized frame. Thereby, synthesized frames can be 
obtained, in which deterioration of image quality is reduced 

20 for each component. Accordingly, a synthesized frame, 
comprising synthesized frames for each component, is enabled 
to be obtained. 

Note that the first through tenth methods for 
synthesizing moving images may be provided as programs that 

25 cause a computer to execute the methods. The programs may be 
recorded on computer readable media. 
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A skilled artisan would know that computer readable media 
are not limited to any specific type of storage device and 
includes any kind of device, including but not limited to: CD' s, 
floppy disks, RAM' s, ROM's, hard disks, magnetic tapes, and 
5 internet downloads, in which computer instructions can be 
stored and/or transmitted. Transmission of computer code 
through a network or through wireless transmission means is also 
within the scope of this invention. Additionally, computer 
code includes, but is not limited to: source, object, and 
10 executable code, and can be in any language, including higher 
level languages, assembly language, and machine language. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a block diagram illustrating the schematic 
construction of amoving image synthesizing apparatus according 
15 to a first embodiment of the present invention. 

Figures 2A, 2B, 2C, and 2D are diagrams for explaining 
a correspondent relationship between a reference frame FrN and 
a frame FrN+1. 

Figure 3 is a diagram for explaining deformation of a 

20 patch. 

Figures 4A and 4B illustrate patches having different 
numbers of rectangular regions therein. 

Figure 5 is a diagram for explaining a correspondent 
relationship between a reference patch P0 and a patch PI. 
25 Figure 6 is a diagram for explaining bilinear insertion. 

Figure 7 is a diagram for explaining assignment of the 
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frame FrN+1 to an integrated image. 

Figure 8 is a diagram for explaining calculation of pixel 
values in an integer coordinate space of the integrated image. 

Figure 9 is a flow chart illustrating the processes 
5 performed by the moving image synthesizing apparatus of the 
first embodiment. 

Figures 10A and 10B are diagrams for explaining the 
processes performed by a moving image synthesizing apparatus 
according to a second embodiment of the present invention. 
10 Figure 11 is a block diagram illustrating the schematic 

construction of amoving image synthesizing apparatus according 
to a third embodiment of the present invention. 

Figure 12 is a flow chart illustrating the processes 
performed by the moving image synthesizing apparatus of the 
15 third embodiment. 

Figures 13A and 13B are diagrams for explaining the 
processes performed by a moving image synthesizing apparatus 
according to a fourth embodiment of the present invention (part 
1) . 

20 Figure 14 is a diagram for explaining the processes 

performed by the moving image synthesizing apparatus according 
to the fourth embodiment of the present invention (part 2) . 

Figure 15 is a flow chart illustrating the processes 
performed by the moving image synthesizing apparatus according 
25 to the fourth embodiment of the present invention. 

Figure 16 is a diagram for explaining other processes 
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performed by the moving image synthesizing apparatus of the 
fourth embodiment (part 1) . 

Figure 17 is a diagram for explaining other processes 
performed by the moving image synthesizing apparatus of the 
5 fourth embodiment (part 2) . 

Figure 18 is a graph illustrating the relationship 
between numbers of rectangular regions and a threshold value 
in the fourth embodiment. 

Figure 19 is a block diagram illustrating the schematic 
10 construction of a moving image synthesizing apparatus according 
to a fifth embodiment of the present invention. 

Figure 20 is a flow chart illustrating the processes 
performed by the moving image synthesizing apparatus of the 
fifth embodiment. 
15 Figure 21 is a flow chart illustrating the processes 

performed by a moving image synthesizing apparatus according 
to a sixth embodiment of the present invention. 

Figure 22 is a graph illustrating the relationship 
between numbers of rectangular regions and a threshold value 
20 in the sixth embodiment . 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Hereinafter, the embodiments of the present invention 
will be described with reference to the attached drawings. 
Figure 1 is a block diagram illustrating the schematic 
25 construction of a moving image synthesizing apparatus according 
to a first embodiment of the present invention. As shown in 
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Figure 1, the moving image synthesizing apparatus of the first 
embodiment comprises: a sampling means 1; a correspondent 
relationship estimating means 2; a coordinate converting means 
3; a correlative value calculating means 4; a comparing means 
5 5; and a synthesizing means 6. The sampling means 1 samples 
a plurality of frames from moving image data MO, which is input 
to the apparatus. The correspondent relationship estimating 
means 2 estimates correspondent relationships among pixels of 
a first frame from among the plurality of frames, which is 

10 designated as a reference frame, and those of a second frame 
from among the plurality of frames. The coordinate converting 
means 3 coordinate converts the second frame to a coordinate 
space of the reference frame, based on the correspondent 
relationships estimated by the correspondent relationship 

15 estimating means 2. The coordinate converting means 3 thus 
obtains a plurality of coordinate converted frames FrTO, FrTl , 
corresponding to numbers of rectangular regions within patches 
(to be described later) for which correspondent relationships 
are estimated. The correlative value calculating means 4 

20 calculates a plurality of correlative values, which represent 
degrees of correlation among the coordinate converted frames 
FrTO, FrTl, and the reference frame, corresponding to the 
numbers of rectangular regions within the patches (to be 
described later) for which correspondent relationships are 

25 estimated. The comparing means 5 compares the degrees of 
correlation among the numbers of rectangular regions, based on 
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the plurality of correlative values. The synthesizing means 
6 generates a synthesized frame FrG having a higher resolution 
than either the reference frame or the second frame, by 
administering interpolation calculations on the reference 
5 frame and the second frame, based on the correspondent 
relationships, which were estimated by the correspondent 
relationship estimating means 2 regarding the frames having the 
number of rectangular regions therein that yielded the highest 
degree of correlation. Note that in the first embodiment, the 

10 synthesized frame FrG has two times the number of pixels of each 
of the sampled frames in both the vertical and horizontal 
directions. Note also that hereinafter, the synthesized frame 
FrG will be described as that which has two times the number 
of pixels of each of the sampled frames. However, the 

15 synthesized frame FrG may have n times (wherein n is a whole 
number) the number of pixels of each of the sampled frames. 

The sampling means 1 samples a plurality of frames from 
the moving image data MO. In the first embodiment, the sampling 
means 1 samples two frames, FrN and FrN+1, from the moving image 

2 0 data MO. Note that the frame FrN is designated as the reference 
frame. Here, the moving image data MO represents a color moving 
image, and the frames FrN and FrN+1 comprise luminance 
chrominance components Y, Cb, and Cr. In the following 
description, processes are performed with respect to each of 

25 the components Y, Cb, and Cr. However, the same processes are 
performed with respect to all of the components. Therefore, 
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a detailed description will be given only for the processes 
administered on the luminance component Y, and descriptions 
regarding the processes administered on the chrominance 
components Cb and Cr will be omitted. 
5 The correspondent relationship estimating means 2 

estimates the correspondent relationship between the reference 
frame FrN and the second frame FrN+1 in the following manner. 
Figures 2A, 2B, 2C, and 2D are diagrams for explaining the 
estimation of the correspondent relationship between the 

10 reference frame FrN and the frame FrN+1. In Figures 2A, 2B, 
2C, and 2D, it is assumed that a circular subject, which is 
included in the reference frame FrN, has moved slightly to the 
right of the figures in the frame FrN+1. 

First, the correspondent relationship estimating means 

15 2 provides a reference patch P0, which has a plurality of 
rectangular regions therein, in the reference frame FrN. 
Figure 2A shows a state in which the reference patch P0 is 
provided in the reference frame FrN. As shown in Figure 2A, 
in the first embodiment, the reference patch P0 comprises 

20 rectangular regions in a 2 x 2 arrangement. Then, a second patch 
PI, which is the same as the reference patch P0, is provided 
at an appropriate position in the frame FrN+1, as shown in Figure 
2B. A correlative value, which represents the degree of 
correlation between the image within the reference patch P0 and 

25 the image within the patch PI, is calculated. The correlative 
value may be calculated as a mean square error according to 
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Formula (1) below. In addition, coordinate axes are set such 
that the horizontal direction in the figures is the x axis, and 
the vertical direction in the figures is the y axis. 
Formula (1) 



wherein E : correlative value 

pi, qi : pixel values of the corresponding pixels 
within the reference patch PO and the patch PI, respectively 
N: the number of pixels within the reference patch 

10 PO and the patch PI 

Next, the patch PI is moved in four directions (up, down, 
left, and right) for a predetermined number of pixels ±Ax and 
± Ay. Correlative values, which represent the degrees of 
correlation between the images within the moved patch PI and 

15 the reference patch PO in the reference frame, are calculated. 
Each correlative value is designated as E(Ax, 0), E(-Ax, 0) , 
E(0, Ay), and E(0, -Ay). 

Then, a gradient direction, in which the correlative 
values E(Ax, 0), E(-Ax, 0), E(0, Ay), and E(0, -Ay) decrease 

20 (that is, in which the degrees of correlation increase) , is 
derived as a correlation gradient. The patch PI is moved by 
a real number multiple in the direction of the correlation 
gradient, as shown in Figure 2C. Specifically, coefficients 
C(Ax, 0), C(-Ax, 0), C(0, Ay), and C(0, -Ay) are calculated 

25 according to Formula (2) below. Then, correlation gradients 
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gx and gy are calculated according to Formulas (3) and (4) , based 
on the coefficients C (Ax, 0) , C (-Ax, 0) , C (0, Ay) , and C (0, -Ay) . 
Formula (2) 



5 Formula (3) 

c(Ax,0)-c(-Ax,0) 
2 

Formula (4) 

c(0,A>Q-c(0-Ay) 
^ 2 

Thereafter, the entire patch PI is moved (Algx, Algy) , 
10 based on the calculated correlation gradients gx and gy. Then, 
the patch PI is repetitively moved until it converges at a 
certain position, as shown in Figure 2D, by repeating the 
processes described above. Here, Al is a parameter that 
determines the speed of convergence, and is a real number. Note 
15 that if the value of Al is set too high, the solutions of the 
repetitive processes will become dispersed. Therefore, it is 
necessary to select an appropriate value (10, for example) 
therefor . 

Further, a lattice point of the patch PI is moved in the 
20 four directions along the coordinate axes, for the distance of 
a predetermined number of pixels . At this time, the rectangular 
regions that contain the moved lattice point become deformed 
as shown in Figure 3, for example. Then, correlative values, 
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which represent the degrees of correlation between the deformed 
rectangular regions and the rectangular regions of the 
reference patch PO corresponding thereto, are calculated. 
Each correlative value is designated as El (Ax, 0) , El (-Ax, 0) , 
5 E1(0, Ay), and El (0, -Ay). 

Then, a gradient direction, in which the correlative 
values El (Ax, 0), El (-Ax, 0), E1(0, Ay), and El (0, -Ay) decrease 
(that is, in which the degrees of correlation increase) , is 
derived in the same manner as above. The lattice points of the 
10 patch PI are moved by a real number multiple in the gradient 
direction. This is performed for all of the lattice points of 
the patch PI, to complete one cycle of a process. The process 
is repeated until the coordinates of the lattice points 
converge . 

15 The amount of movement and the amount of deformation of 

the patch PI with respect to the reference patch P0 is derived 
by the above processes. The correspondent relationships among 
the pixels of the reference patch P0 and the patch PI can be 
estimated, based on these values. 

20 Next, as shown in Figure 4A, the correspondent 

relationship estimating means 2 provides a reference patch P01 
and a second patch Pll, having a greater number of rectangular 
regions therein (4 x 4 in this case) at appropriate positions 
in the reference frame FrN and the second frame FrN+1, 

25 respectively. Then, the amount of movement and the amount of 
deformation of the second patch Pll with respect to the 
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reference patch P01 is derived in the same manner as that 
described above. Thereafter, the correspondent relationships 
among the pixels of the reference patch P01 and the second patch 
Pll are estimated. 
5 Note that as shown in Figure 4B, a reference patch P02 

and a second patch P12 having a still greater number of 
rectangular regions therein (8 x 8 in this case) maybe employed. 
However, in the first embodiment, the correspondent 
relationships are estimated for rectangular regions of the two 

10 steps of 2 x 2 and 4x4 arrangements. In the first embodiment, 
the correspondent relationships, which have been estimated for 
the reference patch P0 and the second patch PI having 
rectangular regions therein in a 2 x 2 arrangement, are referred 
to as a first correspondent relationship. The correspondent 

15 relationships, which have been estimated for the reference 
patch P01 and the second patch Pll having rectangular regions 
therein in a 4 x 4 arrangement, are referred to as a second 
correspondent relationship . 

The coordinate converting means 3 converts the second 

20 frame FrN+1 into the coordinate space of the reference frame 
FrN for each number of rectangular regions, based on the first 
and second correspondent relationships, to obtain coordinate 
converted frames FrTO and FrTl, in the following manner . First, 
the obtainment of the coordinate converted frame FrTO, based 

25 on the first correspondent relationship, will be described. 
Note that in the following description, conversion and 
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synthesis is performed only on the regions of the reference 
frame FrN within the reference patches PO and P01. Similarly, 
conversion and synthesis is performed only on the regions of 
the second frame FrN+1 within the second patches PI and Pll. 
5 In the first embodiment, the coordinate conversion is 

performed employing bilinear conversion. The coordinate 
conversion by bilinear conversion is defined by the following 
Formulas (5) and (6) . 
Formula (5) 

10 x = (1 - w)(l - v)xl + (1 - v)ux2 + (1 - u)vx3 + uvx4 

Formula (6) 

y = (1 - w)(l - v)y\ + (1 - v)uy2 + (1 - u)vy3 + uvyA 

Formulas (5) and (6) interpolate coordinates within the 
patch PI, which is defined by four points (xn, yn) (wherein 

15 l<n<4) within two dimensional coordinates, by a normalized 
coordinate system (u, v) (wherein 0<u, v<l) . Coordinate 
conversion within two desired rectangular shapes are enabled 
by combinations of Formulas (5) and (6) , and inverse conversion 
of Formulas (5) and (6) . 

20 Here, as shown in Figure 5, the position that a point (x, 

y) within the patch PI (xn, yn) corresponds to within the 
reference patch PO (x'n, y'n) is considered. First, normalized 
coordinates (u, v) are derived for the point (x, y) within the 
patch PI (xn, yn) . The normalized coordinates (u, v) are 

25 derived by inverse conversions of Formulas (5) and (6) . Then, 
coordinates (x' , y' ) , which correspond to the point (x, y) are 



derived, based on the reference patch PO (x'n, y'n) that 
corresponds to (u, v) , by Formulas (5) and (6). Here, there 
are cases that the point (x' , y' ) is defined by real number 
coordinates where pixel values do not originally exist, whereas 
5 the point (x, y) is defined by integer coordinates, where pixel 
values exist. Therefore, the pixel value of the converted 
integer coordinates is derived as a weighted sum of the 
coordinates (x' , y' ) , within a region defined by the eight 
adjacent integer coordinates surrounding integer coordinates 

10 of the reference patch PO. 

Specifically, as shown in Figure 6, a pixel value of 
integer coordinates b(x, y) is calculated, based on a pixel 
value of the frame FrN+ 1 , which has been converted within a 
region surrounded by eight adjacent integer coordinate points 

15 b(x-l, y-1), b(x, y-1), b(x+l, y-1), b(x-l, y) , b(x+l, y) , b(x-l, 
y+1) , b(x, y+1, andb(x, y+1). Here, an m number of pixel values 
of the frame FrN+1 are converted Within the region surrounded 
by the eight pixels. If the pixel values of each of the 
converted pixels are set as Itj(x°, y°) (wherein l<j<m) , then 

20 the pixel value It(x~, y"), at the integer coordinates b(x, y) 
can be calculated by Formula (7) below. Note that in Formula 
(7) , (pis a function that represents a weighted sum calculation. 

25 
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Formula (7) 

It(x*,y*) = <p{Itj(x°,y°)) 

{W\ x It\(x°, y°) + W2x It2(x°, y°) + -- Wm x Itm(x°, y°)} 
(W\ + W2 + ■ ■ Wk) 

fwjxltj(x°,y°) 

m 

wherein Wj (l^j^m) : the product of division ratios within the 
coordinates as viewed from the integer pixels in the vicinity 
of the position that the pixel value Itj(x°, y° ) is assigned 
to. Here, for simplicity' s sake, a case is considered in which 
two pixel values Itl and It2 are converted to the; region 
surrounded by eight pixels, employing Figure 6. In this case, 
the pixel value It(x^, y A ) at the integer coordinates b(x, y) 
can be calculated by Formula (8) below. 
Formula (8) 

//(x A , y A ) = = (Wl x Itl + W2 + Itl) 

K y } W\ + W2 y } 

wherein Wl=u x v, and W2=(l-s) x (1-t) . 

By performing the above processes for every pixel within 
the patch PI, the image within the patch PI is converted into 
the coordinate space of the reference frame FrN, and a 
coordinate converted frame FrTO is obtained. 

A coordinate converted frame FrTl is obtained by 
converting the image within the patch Pll into the coordinate 
space of the reference frame FrN in a similar manner, based on 
the second correspondent relationship. 
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The correlative value calculating means 4 calculates a 
correlative value dO between the reference frame FrN and the 
coordinate converted frame FrTO, and a correlative value dl 
between the reference frame FrN and the coordinate converted 
frame FrTl. The correlative value dO is calculated as shown 
in Formula (9) below. Specifically, the sums of the absolute 
values of the differences between the pixel values FrN (x, y) 
and FrTO (x, y) , for corresponding pixels within the reference 
frame FrN and the coordinate converted frame FrTO, are divided 
by the number of pixels m, with respect to all of the pixels 
within the reference patch PO . The mean value of the quotients, 
that is, the absolute values of the differences, is calculated 
as the correlative value dO . In addition, the correlative value 
dl is calculated as shown in Formula (10) . Specifically, the 
mean value of the absolute values of the differences between 
the pixel values FrN(x, y) and FrTl (x, y) , for corresponding 
pixels within the reference frame FrN and the coordinate 
converted frame FrTl, are calculated with respect to all of the 
pixels within the reference patch PO, as the correlative value 
dl . Note that the correlative values dO and dl decrease as the 
degree of correlation between the reference frame FrN and the 
coordinate converted frames FrTO and FrTl increase. 

Formula (9) 

£\FrT0(x,y)-FrN(x,y) 

d0 = 

m 
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Formula (10) 

m 

^r\FrTl(x,y)-FrN(x y y) 

dl = 

m 

Note that in the first embodiment, the correlative values 
dO and dl are calculated from the absolute values of the 
5 differences in pixel values of the pixels that correspond among 
the reference frame FrN and the coordinate converted frames FrTO 
and FrTl . Alternatively, the correlative values may be 
calculated from squares of the differences. In addition, the 
correlative values may be the sums of the absolute values of 

10 the differences in pixel values of the pixels that correspond 
among the reference frame FrN and the coordinate converted, 
frames FrTO and FrTl . As a further alternative, histograms may 
be calculated for the reference frame FrN, the coordinate 
converted frame FrTO, and the coordinate converted frame FrTl. 

15 In this case, a mean value, a median value, a differential value 
of the standard deviation, or a cumulative sum of the 
differential values of the histograms may be employed as the 
correlative value. As a still further alternative, motion 
vectors of the coordinate converted frames FrTO and FrTl with 

20 respect to the reference frame FrN may be calculated for each 
pixel or each small region of the reference frame FrN. In this 
case, the mean value, the mediam value, or the standard 
deviation of the motion vectors may be employed as the 
correlative value. In addition, the cumulative sum of the 

25 histograms of the motion vectors may be employed as the 
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correlative value . 

The comparing means 5 compares the correlative value dO 
and the correlative value dl, and selects the smaller 
correlative value, that is, the correlative value that 
5 represents the greater degree of correlation. Note that in the 
first embodiment, it is assumed that the correlative value dl, 
which was calculated based on the reference patch P01 and the 
patch Pll having rectangular regions therein at a 4 x 4 
arrangement, is selected. 

10 The synthesizing means 6 generates the synthesized frame 

FrG, by administering interpolation calculations on the 
reference frame FrN and the frame FrN+1 , based on the second 
correspondent relationship dl, which was estimated based on the 
reference patch P01 and the patch Pll having the rectangular 

15 regions therein in a 4 x 4 arrangement. Specifically, as shown 
in Figure 7, first, an integrated image having an ultimately 
necessary number of pixels (in the first embodiment, a case is 
described in which the integrated image has two times the number 
of pixels of the frames FrN and FrN+1 in both the horizontal 

20 and vertical directions; however, the number of pixels may be 
n times (wherein n is a whole number) the number of pixels of 
the frames FrN and FrN+1) is prepared. Then, pixel values of 
the pixels of the frame FrN+1 (the region within the patch Pll) 
are assigned to the integrated image, based on the second 

25 correspondent relationship derived by the correspondent 
relationship estimating means 2. If the assignment function 
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is designated as n, the pixel values of each pixel within the 
frame FrN+1 is assigned to the integrated frame according to 
Formula (11) below. 
Formula (11) 
5 I1N + y°) = U(FrN + y)) 

wherein I1N+1 (x° , y° ) : pixel values of the frame FrN+1 which 

have been assigned to the integrated frame; and 

FrN+1 (x, y) : pixel values of the frame FrN+1 
A first interpolation frame having pixel values of II (x° , 
10 y°) ( = I1N+I(x°, y°)) for each pixel is obtained, by assigning 
pixel values of the frame FrN+1 to the integrated image in this 
manner . 

During assignment of the pixel values to the integrated 
image, there are cases in which pixels of the frame FrN+1 do 

15 not correspond to the integer coordinates (that is, the 
coordinates where a pixel value should exist) of the integrated 
image, depending on the relationship between the number of 
pixels in the integrated image and the number of pixels in the 
frame FrN+1. In the first embodiment, the pixel values at 

20 integer coordinates within the integrated image are derived in 
the following manner. A region of eight integer coordinates 
that surround the integer coordinate is set. Then, the pixel 
value of the integer coordinate within the region is derived 
as a weighted sum of the pixel values of the pixels from the 

25 frame FrN+1, which are assigned within this region. 

That is, as shown in Figure 8, the pixel value for an 
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integer coordinate p (x, y) within the integrated image is 
calculated based on the pixel values from the frame FrN+1, which 
have been assigned within a region surrounded by integer 
coordinates p(x-l, y-1), p(x, y-1) , p(x+l, y-1), p(x-l, y) , 
5 p(x+l, y) , p(x-l, y+1), p (x, y+1), and p(x+l, y+1). Here, k 
number of pixel values are assigned from the frame FrN+1 to the 
region surrounded by the eight pixels. If the pixel values of 
each of the assigned pixels are designated as IlN+li (x°, 
y°) (wherein l<i<k) , the pixel value I1N+1 (x^, y* ) of the integer 
10 coordinates can be calculated by Formula (12) below. Note that 
in Formula (12), $ is a function that represents weighted sum 
calculation. 

Formula (12) 

UN + l(x A , j/ A ) = <D(7 IN + y°)) 

= { (Ml x UN + 1 1(jc°, y°) + M2x UN + 1 2(x°, y °) + - ■ + Mk x UN + lk(x°, y°)) } 

(A/1+A/2 + ---+JW*) 

k 

^(MixIlN-hli(x 0 7 y°) 
_ _i=i 

15 

wherein Mi(l<i<k): the product of division rations within 
coordinates at positions where pixel values IlN+li(x°, y°) are 
assigned, when viewed from integer pixels in their vicinity. 

Here, for the sake of simplicity, a case in which two pixel 
20 values I1N+11 and I1N+12 are assigned from the frame FrN+1 to 
the region surrounded by eight pixels shown in Figure 8. In 
this case, the pixel value HN+lfx^, y*) at the integer 
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coordinate p (x, y) can be calculated by Formula (13) below. 
Formula (13) 

7W + l(x A ,^ A ) = = (MlxIlN + ll+M2xIlN + 12) 

Ml +M2 

wherein Ml=u x v; and M2=(l-s) x (1-t) 
5 Pixel values I1N+1 (x A , y A ) for all of the integer 

coordinates within the integrated image are obtained by 
assigning the pixel values of the frame FrN+1 thereto. In this 
case, the pixel values II (x°, y°) of the first interpolation 
frame are I1N+1 (x°, y°) . 

10 Note that regarding the reference frame FrN, the pixels 

thereof are interpolated and directly assigned to the integer 
coordinates of the integrated image. The pixel values obtained 
by assigning the reference frame FrN to the integer coordinates 
of the integrated image are designated as 12 (x A , y A ) . The frame 

15 having the pixel values 12 (x A , y A ) is referred to as a second 
interpolation frame . 

The synthesized frame FrG, which has pixel values FrG(x A , 
y A ) , is generated by adding or weighting and adding the pixel 
values II (x A , y A ) and 12 (x A , y A ) of corresponding pixels of the 

20 first and second interpolation frames. Note that the 
synthesized frame FrG, having the pixel values FrG(x A , y A ), may 
be generated from only the pixel values II (x A , y A ) , 

There are cases in which pixel values cannot be assigned 
to all of the integer coordinates of the integrated images. In 

25 these cases, the pixel values for the integer coordinates, for 
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which no pixel values are assigned, may be calculated from 
assigned pixel values or from pixel values of the reference 
frame that correspond to the coordinates, for which no pixel 
values are assigned. The calculation may be an interpolation 
5 calculation such as linear interpolation and spline 
interpolation . 

The process described above derives the synthesized frame 
FrG with respect to the luminance component Y. Synthesized 
frames are obtained in the same manner, for the chrominance 

10 components Cb and Cr as well. A final synthesized frame is 
obtained by synthesizing the synthesized frame FrG ( Y) derived 
from the luminance component Y, a synthesized frame FrG(Cb) 
derived from the chrominance component Cb, and a synthesized 
frame FrG(Cr) , derived from the chrominance component Cr. Note 

15 that it is preferable to estimate the correspondent 
relationships between the reference frame FrN and the frame 
FrN+1 only for the luminance component Y. Then the processes 
are administered regarding the chrominance components Cb and 
Cr, based on the correspondent relationships, which have been 

20 estimated for the luminance component Y. This expedites the 
synthesis process . 

Next, the operation of the first embodiment will be 
described. Figure 9 is a flow chart illustrating the processes 
performed by the moving image synthesizing apparatus of the 

25 first embodiment. First, moving image data MO is input to the 
sampling means 1 (step SI) , and the reference frame FrN and the 
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frame FrN+1 are sampled therefrom (step S2) . Next, a plurality 
of correspondent relationships between the reference frame FrN 
and the frame FrN+1, corresponding to the numbers of rectangular 
regions within the patches, are estimated by the correspondent 
5 relationship estimating means 2 (step S3) . 

Then, the frame FrN+1 is converted into the coordinate 
space of the reference frame FrN by the coordinate converting 
means 3, based on the plurality of correspondent relationships, 
to obtain a plurality of coordinate converted frames, FrTO and 

10 FrTl (step S4) . Thereafter, the correlative value dO, which 
represents the degree of correlation between the reference 
frame FrN and the coordinate converted frame FrTO, and the 
correlative value dl, which represents the degree of 
correlation between the reference frame FrN and the coordinate 

15 converted frame FrTl, are calculated by the correlative value 
calculating means 4 (step S5) . Further, the correlative value 
that represents the larger degree of correlation is selected 
by the comparing means 5 (step S6) . 

Finally, the synthesizing means generates the 

20 synthesized frame FrG from the reference frame FrN and the frame 
FrN+1, based on the correspondent relationship (here, the 
second correspondent relationship) which was estimated for the 
number of rectangular regions (here, 4x4) that yielded the 
selected correlative value dl (step S7) , and the process ends. 

25 In this manner, the synthesized frame FrG is generated 

from the reference frame FrN and the frame FrN+1, based on the 
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correspondent relationship which was estimated for the number 
of rectangular regions that yielded the largest degree of 
correlation in the first embodiment. Therefore, the 
synthesized frame FrG is enabled to be generated employing the 
5 optimal number of rectangular regions for following the 
movement and/or deformation of a moving subject within the 
frames. Accordingly, a synthesized frame FrG having high image 
quality is enabled to be obtained, regardless of movement of 
a subject included in the frames. 

10 Next, a second embodiment of the present invention will 

be described. In the first embodiment described above, the 
synthesized frame FrG is generated based on the correspondent 
relationship, estimated for the number of rectangular regions 
that yields the largest degree of correlation, of the entirety 

15 of the images in the patches within the reference frame FrN and 
the frame FrN+1. In the second embodiment, correlative values 
are calculated for each rectangular region within the patches, 
and a synthesized frame FrG is generated based on the 
correspondent relationships of each of the rectangular regions . 

20 Figures 10A and 10B are diagrams for explaining the 

calculation of correlative values for each of the rectangular 
regions. As shown in Figure 10A, each rectangular region of 
a reference patch P0 and a second patch PI, in which rectangular 
regions are provided in a 2 x 2 arrangement, is designated as 

25 rectangular regions Al, A2, A3, and A4 . Meanwhile, each 
rectangular region of a reference patch P0 and a second patch 
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PI, in which rectangular regions are provided in a 4 x 4 
arrangement, is designated as A11-A14, A21-A24, A31-A34, and 
A41-A44. Note that the rectangular region Al corresponds to 
the rectangular regions All through A14, the rectangular region 
5 A2 corresponds to the rectangular regions A21 through A2 4, the 
rectangular region A3 corresponds to the rectangular regions 
A31 through A34, and the rectangular region A4 corresponds to 
the rectangular regions A41 through A44. 

In the case that the rectangular regions are provided in 

10 the 2x2 arrangement, correlative values for the reference 
frame FrN and the frame FrN+1 are calculated for each of the 
rectangular regions Al through A4 . The correlative values 
calculated for each of the rectangular regions Al through A4 
are designated as correlative values dOl, d02, d03, and d04, 

15 respectively. 

On the other hand, in the case that the rectangular regions 
are provided in the 4x4 arrangement, correlative values are 
calculated for each of the rectangular regions All through A14, 
A21 through A2 4, A31 through A3 4, and A41 through A4 4. The mean 

20 value of the correlative values calculated for the rectangular 
regions All through A14 is calculated and designated as 
correlative value dll . The mean value of the correlative values 
calculated for the rectangular regions A21 through A24 is 
calculated and designated as correlative value dl2. The mean 

25 value of the correlative values calculated for the rectangular 
regions A31 through A34 is calculated and designated as 
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correlative value dl3. The mean value of the correlative values 
calculated for the rectangular regions A41 through A44 is 
calculated and designated as correlative value dl4. 

Then, the correlative values of corresponding regions are 
5 compared. That is, the correlative value dOl of the rectangular 
region Al is compared against the mean correlative value dll 
of the rectangular regions All through A14, the correlative 
value d02 of the rectangular region A2 is compared against the 
mean correlative value dl2 of the rectangular regions A21 

10 through A24, the correlative value d03 of the rectangular region 
A3 is compared against the mean correlative value dl3 of the 
rectangular regions A31 through A34, and the correlative value 
d04 of the rectangular region A4 is compared against the mean 
correlative value dl4 of the rectangular regions A41 through 

15 A44. Then, the synthesized frame FrG is generated from the 
reference frame FrN and the frame FrN+1 for each of the 
rectangular regions, based on the correspondent relationship, 
which was estimated for the number of rectangular regions that 
yielded the higher degree of correlation, based on the 

20 comparison results . 

In the second embodiment, the larger the degree of 
correlation, the lower the correlative value is. Therefore, 
for example, in the case that the correlative value dOl is less 
than the correlative value dll, interpolation calculations are 

25 performed based on the correspondent relationships, which were 
estimated for the rectangular regions provided in the 2x2 
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arrangement, regarding the regions that correspond to the 
rectangular region Al in the reference frame FrN and the frame 
FrN+1. Likewise, in the case that the correlative value d02 
is less than then correlative value dl2, interpolation 
5 calculations are performed based on the correspondent 
relationships, which were estimated for the rectangular regions 
provided in the 2x2 arrangement, regarding the regions that 
correspond to the rectangular region A2 in the reference frame 
FrN and the frame FrN+1. 

10 On the other hand, in the case that the correlative value 

d03 is greater than the correlative value dl3, interpolation 
calculations are performed based on the correspondent 
relationships, which were estimated for the rectangular regions 
provided in the 4x4 arrangement, regarding the regions that 

15 correspond to the rectangular region A3 in the reference frame 
FrN and the frame FrN+1. Further, in the case that the 
correlative value d04 is greater than the correlative value dl4, 
interpolation calculations are performed based on the 
correspondent relationships, which were estimated for the 

20 rectangular regions provided in the 4x4 arrangement, regarding 
the regions that correspond to the rectangular region A4 in the 
reference frame FrN and the frame FrN+1. Finally, the 
synthesized frame FrG, comprising the pixels obtained by the 
interpolation calculations, is generated* 

25 In this manner, the second embodiment estimates the 

correspondent relationships to be employed during generation 
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of the synthesized frame FrG for each rectangular region. 
Thereby, the synthesized frame FrG is generated for each 
rectangular region within the reference patch and the second 
patch. For this reason, the synthesized frame FrG is enabled 
5 to be generated employing the optimal number of rectangular 
regions for following the movement of subjects included in each 
portion within the frame. Accordingly, a synthesized frame FrG 
is enabled to be obtained, regardless of movement of a subject 
included in the frames. 

10 In the first and second embodiments described above, the 

correlative values dO and dl are calculated for each of the 
luminance chrominance components Y, Cb, and Cr . However, a 
single correlative value dO'may be calculated, by weighting a 
correlative value dOY for the luminance component, and 

15 correlative values dOCb and dOCr for the chrominance components 
with weighting coefficients al, bl, and cl, as shown in Formula 
(14) below. 

Formula (14) 

dP= al • diY + bl - diCb + cl • diCr 

20 Alternatively, Euclid distances, weighted by weighting 

coefficients a2, b2, and c2, among luminance components FrTi 
(x, y) and chrominance components FrTiCb (x, y) and FrTiCr (x, 
y) of a coordinate converted frame FrTi, and luminance 
components FrNY (x, y) , and chrominance components FrNCb and 

25 FrNCr, may be calculated for each of the pixels within the 
reference patch P0, as shown in Formula (15) below. The Euclid 
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distances are designated as correlative values di" (x, y) for 
each pixel. When applied to the first embodiment, the mean 
value of the correlative values di" (x, y) for all of the pixels 
within the reference patch PO maybe employed as the correlative 
5 value employed for comparison. When applied to the second 
embodiment, the mean value of the correlative values di" (x, 
y) for all of the pixels within each rectangular region may be 
employed as the correlative value employed for comparison. 
Formula (15) 

] df(x,y) = {a2(FrTiY(x,y)-FrNY(x,y)) 2 + b2(FrTiCb(x,y)- FrNCb(x,y)) 2 

+ c2(FrTiCr(x,y)-FrNCr(x,y)) 2 } 05 

Note that in the first and second embodiments described 
above, the synthesized frames FrG are generated from two frames 
FrN and FrN+1. However, a synthesized frame FrG may be 
generated from three or more frames. For example, a case is 

15 considered in which a synthesized frame FrG is generated from 
T number of frames FrN+t' (wherein 0<t'<T-l) . In this case, 
a plurality of correspondent relationships, between a reference 
frame FrN (=FrN+0) and each of a plurality of other frames FrN+t 
(wherein l<t<T-l), are estimated, according to the numbers of 

20 rectangular regions within patches thereof. Further, a 
plurality of coordinate converted frames are obtained, 
according to the plurality of correspondent relationships. 
Then, correlative values are calculated between the reference 
frame FrN and each of the plurality of coordinate converted 

25 frames. Then, the correspondent relationship estimated for 
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the number of rectangular regions, to be employed in the 
generation of a synthesized frame is selected, based on the 
correlative values. Next, a synthesized frame is generated, 
based on the correspondent relationship, which was estimated 
5 for the selected number of rectangular regions, in the same 
manner as in the first embodiment. This synthesized frame is 
designated as an intermediate synthesized frame FrGt . Note 
that pixel values of the intermediate synthesized frame FrGt 
at integer coordinates of an integrated images are designated 
10 as FrGt (x A , y A ) . 

Intermediate synthesized frames FrGt are obtained for all 
of the other frames FrN+t. Finally, a synthesized frame FrG 
comprising pixel values (x^, y A ) is generated, by adding 
corresponding pixels of the intermediate frames according to 
15 formula (16) below. 

Formula (16) 
r-i 
t=\ 

There are cases in which pixel values cannot be assigned 
to all of the integer coordinates of the integrated images. In 

20 these cases, the pixel values for the integer coordinates, for 
which no pixel values are assigned, may be calculated from 
assigned pixel values or from pixel values of the reference 
frame that correspond to the coordinates, for which no pixel 
values are assigned. The calculation may be an interpolation 

25 calculation such as linear interpolation and spline 
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interpolation. 

In the case that the synthesized frame FrG is generated 
from three or more frames, the intermediate synthesized frames 
may be generated in the same manner as that described in the 
5 second embodiment. That is, correlative values may be 
calculated for each of the rectangular regions within the 
patches, then the intermediate synthesized frames may be 
generated based on the correspondent relationships of each of 
the rectangular regions. 

10 Next, a third embodiment of the present invention will 

be described. Figure 11 is a block diagram illustrating the 
schematic construction of a moving image synthesizing apparatus 
according to the third embodiment of the present invention. The 
moving image synthesizing apparatus of the third embodiment 

15 comprises: a sampling means 11; a correspondent relationship 
estimating means 12; a coordinate converting means 13; a 
correlative value calculating means 14; a comparing means 15; 
and a synthesizing means 16, instead of the sampling means 1; 
the correspondent relationship estimating means 2; the 

20 coordinate converting means 3; the correlative value 
calculating means 4; the comparing means 5; and the synthesizing 
means 6 of the first embodiment. The coordinate converting 
means 13 coordinate converts frames other than a reference frame 
to a coordinate space of the reference frame, based on the 

25 correspondent relationships estimated by the correspondent 
relationship estimating means 12. The coordinate converting 
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means 13 thus obtains a plurality of coordinate converted frames 
FrTi. The correlative value calculating means 14 calculates 
a plurality of correlative values di, which represent degrees 
of correlation among the coordinate converted frames FrTi and 
5 the reference frame. The comparing means 15 compares the 
correlative values di against a threshold value Thl . The 
comparing means 15 causes the correspondent relationship 
estimating means 12, the coordinate converting means 13, and 
the correlative value calculating means 14 to increase the 

10 number of rectangular regions within patches, for which 
correspondent relationships are estimated, in a stepwise manner , 
to estimate correspondent relationships, to obtain coordinate 
converted frames FrTi, and to calculate correlative values di 
for each increased step of the number of rectangular regions, 

15 until the correlative value becomes less than or equal to the 
threshold value Thl (to be described later) . The synthesizing 
means 16 generates a synthesized frame FrG having a higher 
resolution than the sampled frames, by administering 
interpolation calculations on the reference frame and the other 

20 frame, based on the correspondent relationships, which were 
estimated by the correspondent relationship estimating means 
12 regarding the frames having the number of rectangular regions 
therein that yielded the correlative value di less than or equal 
to the threshold value Thl. 

25 The sampling means 11 samples a plurality of frames from 

moving image data MO in the same manner as the sampling means 
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1 of the first embodiment. In the third embodiment as well, 
the sampling means 11 samples two frames, FrN and FrN+1, from 
the moving image data MO . 

In the first embodiment, the correspondent relationship 
estimating means 2 estimated the first correspondent 
relationship, then estimated the second correspondent 
relationship after increasing the number of rectangular regions 
in a stepwise manner. In the third embodiment, the 
correspondent relationship estimating means 12 derives a first 
correspondent relationship, then estimates further 
correspondent relationships after increasing the number of 
rectangular regions, as will be described later. 

The coordinate converting means 13 obtains a coordinate 
converted frame FrTO by employing bilinear conversion, as 
illustrated in Formulas (5) and (6) , to coordinate convert the 
frame FrN+1 to the coordinate space of the reference frame FrN, 
based on the estimated first correspondent relationship. Note 
that in the following description, conversion and synthesis is 
performed only for the regions within a reference patch PO of 
the reference frame FrN and a patch PI within the other frame 
FrN+ 1 . 

The correlative value calculating means 14 calculates the 
correlative value dO, which represents the degree of 
correlation between the reference frame FrN and the coordinate 
converted frame FrTO, according to Formula (9). 

The comparing means 15 compares the correlative value dO 
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against the threshold value Thl. The threshold value Thl is 
predetermined and recorded in a memory (not shown) of the 
comparing means 15. Note that a value on the order of 10 may 
be employed as the threshold value Thl, in the case that the 
moving image data MO is 8 bit data. In the case that the 
correlative value dO is less than or equal to the threshold value 
Thl, the number of rectangular regions within the patches P0 
and PI, for which the correlative value dO was calculated, is 
determined. This number of rectangular regions is designated 
as the number of rectangular regions for which the correspondent 
relationship, to be employed by the synthesizing means 16 to 
generate the synthesized frame FrG from the frames FrN and FrN+1, 
was estimated. 

In the case that the correlative value dO exceeds the 
threshold value Thl, the comparing means 15 controls the 
correspondent relationship estimating means 12, the coordinate 
converting means 13, and the correlative value calculating 
means 14 to increase the number of rectangular regions within 
patches, for which correspondent relationships are estimated, 
one step in a stepwise manner, to estimate correspondent 
relationships, to obtain a coordinate converted frame FrTl, and 
to calculate a correlative value for the increased number of 
rectangular regions . 

Specifically, the correspondent relationship estimating 
means 12 increases the number of rectangular regions within the 
reference patch P0 and the patch PI from a 2 x 2 arrangement 



II 

to a 4 x 4 arrangement, as shown in Figure 4A. The reference 
patch PO and the patch PI are provided at appropriate positions 
in the reference frame FrN and the frame FrN+1, respectively. 
Then, the amount of movement and the amount of deformation of 
5 the patch PI with respect to the reference patch PO is derived 
in the same manner as that described above. Thereafter, the 
correspondent relationships among the pixels of the reference 
patch PO and the patch PI are estimated. Next, the coordinate 
converting means 13 converts the frame FrN+1 to a coordinate 

10 space of the reference frame FrN, based on the correspondent 
relationships estimated by the correspondent relationship 
estimating means 12. The coordinate converting means 13 thus 
obtains a coordinate converted frame FrTl . The correlative 
value calculating means 14 calculates the correlative value dl, 

15 which represents the degree of correlation between the 
coordinate converted frame FrTl and the reference frame FrN, 
according to Formula (10). 

Then, the comparing means 15 compares the calculated 
correlative value dl against the threshold value Thl . In the 

20 case that the correlative value dl exceeds the threshold value 
Thl, the number of rectangular regions within the reference 
patch PO and the patch PI is further increased, as shown in Figure 
4B . Thereafter, the correspondent relationship estimating 
means 12, the coordinate converting means 13 and the correlative 

25 value calculating means 14 are caused to estimate correspondent 
relationships, to obtain coordinate converted frames FrTi 



108 



(wherein -l<i) (hereinafter, the variable "i" will be employed 
for generalization) , and to calculate correlative values di, 
for each stepwise increase in the number of rectangular regions, 
until the correlative value di becomes less than or equal to 
5 the threshold value Thl. Note that in the third embodiment, 
the number of rectangular regions within the reference patch 
PO and the patch pi of the i-th step is 2 i+1 x 2 i+1 . The coordinate 
converted frame, obtained by the number of rectangular regions 
of the i-th step, is designated as coordinate converted frame 

10 FrTi. The correlative value, calculated for the patches having 
therein the number of rectangular regions of the i-th step, is 
designated as correlative value di . The number of rectangular 
regions within the reference patch PO and the patch PI that 
yields the correlative value di which is less than or equal to 

15 the threshold value Thl is determined. This number of 
rectangular regions is designated as the number of rectangular 
regions to be employed to generate the synthesized frame. 

Note that if the number of rectangular regions becomes 
excessively large, each of the rectangular regions becomes too 

20 small. This will cause similar images to be represented in a 
plurality of rectangular images, which in turn may cause an 
inability for the movement and/or deformation of the second 
patch to follow the movement and/or deformation of the subject. 
Therefore, it is preferable that an upper limit is set for the 

25 number of rectangular regions. 

The synthesizing means 16 administers interpolation 
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calculations on the reference frame FrN and the frame FrN+1, 
based on the correspondent relationship, which was estimated 
for the number of rectangular regions designated to be employed 
in the generation of the synthesized frame, and generates the 
5 synthesized frame FrG in the same manner as the synthesizing 
means 6 of the first embodiment. 

Note that in the third embodiment, synthesized frames FrG 
are obtained for the luminance component Y and the chrominance 
components Cb and Cr, as in the first embodiment. A final 

10 synthesized frame is obtained by synthesizing the synthesized 
frame FrG(Y) derived from the luminance component Y, a 
synthesized frame FrG(Cb) derived from the chrominance 
component Cb, and a synthesized frame FrG(Cr) , derived from the 
chrominance component Cr. 

15 Next, the operation of the third embodiment will be 

described. Figure 12 is a flow chart illustrating the processes 
performed by the moving image synthesizing apparatus of the 
third embodiment. First, moving image data MO is input to the 
sampling means 11 (step Sll), and the reference frame FrN and 

20 the frame FrN+1 are sampled therefrom (step S12) . Next, a 
correspondent relationship between the reference frame FrN and 
the frame FrN+1 is estimated by the correspondent relationship 
estimating means 12 (step S13) . 

Then, the frame FrN+1 is converted into the coordinate 

25 space of the reference frame FrN by the coordinate converting 
means 13, based on the estimated correspondent relationships, 
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to obtain the coordinate converted frame FrTO (step S14) . 
Thereafter, the correlative value dO, which represents the 
degree of correlation between the reference frame FrN and the 
coordinate converted frame FrTO, is calculated by the 
5 correlative value calculating means 14 (step S15) . Whether the 
correlative value is less than or equal to the threshold value 
Thl is judged by the comparing means 15 (step S16) . 

In the case that the judgment in step S16 is negative, 
the number of rectangular regions within the reference patch 
10 P0 and the patch PI is increased by one step (step S17) . Then, 
the process returns to step S13, and steps S13 through S16 are 
repeated. 

In the case that the judgment in step S16 is affirmative, 
the number of rectangular regions that yielded the correlative 

15 value is determined to be that which is to be employed in 
generating the synthesized frame (step S18) . Then, the 
synthesizing means 16 generates the synthesized frame FrG from 
the reference frame FrN and the frame FrN+1, based on the 
correspondent relationship which was estimated for the 

20 determined number of rectangular regions (step S19) , and the 
process ends. 

In this manner, the synthesized frame FrG is generated 
from the reference frame FrN and the frame FrN+1, based on the 
correspondent relationship which was estimated for the number 
25 of rectangular regions that yielded the correlative value less 
than or equal to the threshold value Thl , in the third embodiment . 
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Therefore, the synthesized frame FrG is enabled to be generated 
employing the optimal number of rectangular regions for 
following the movement and/or deformation of a moving subject 
within the frames. Accordingly, a synthesized frame FrG having 
5 high image quality is enabled to be obtained, regardless of 
movement of a subject included in the frames. 

Next, the fourth embodiment of the present invention will 
be described. In the third embodiment described above, the 
synthesized frame FrG is generated based on the correspondent 

10 relationship, estimated for the number of rectangular regions 
that yields the correlative value less than or equal to the 
threshold value Thl, of the entirety of the images in the patches 
within the reference frame FrN and the frame FrN+1. In the 
fourth embodiment, correlative values are calculated for each 

15 rectangular region within the patches . Each of the correlative 
values are compared against the threshold value Thl, and a 
synthesized frame FrG is generated based on the correspondent 
relationships of each of the rectangular regions. Note that 
in the fourth embodiment, Formula (9) is employed to calculate 

20 the sums of the absolute values of the differences between the 
pixel values FrN (x, y) and FrTO (x, y) , for corresponding pixels 
within the reference frame FrN and the coordinate converted 
frame FrTO, divided by the number of pixels m, with respect to 
all of the pixels within each of the rectangular regions. The 

25 mean value of the quotients, that is, the absolute values of 
the differences of each pair of corresponding pixels, is 



112 



calculated as the correlative value of each rectangular region. 

Figures 13A and 13B are diagrams for explaining the 
calculation of correlative values for each of the rectangular 
regions. As shown in Figure 13A, each rectangular region of 
5 a reference patch PO and a patch PI, in which rectangular regions 
are provided in a 2 x 2 arrangement, is designated as rectangular 
regions Al, A2, A3, and A4 . Meanwhile, each rectangular region 
of a reference patch PO and a second patch PI, in which 
rectangular regions are provided in a 4 x 4 arrangement, is 

10 designated as A11-A14, A21-A24, A31-A34, andA41~A44, as shown 
in Figure 13B. Note that the rectangular region Al corresponds 
to the rectangular regions All through A14, the rectangular 
region A2 corresponds to the rectangular regions A21 through 
A24, the rectangular region A3 corresponds to the rectangular 

15 regions A31 through A34, and the rectangular region A4 
corresponds to the rectangular regions A41 through A44. 

In the case that the rectangular regions are provided in 
the 2x2 arrangement, correlative values for the reference 
frame FrN and the frame FrN+1 are calculated for each of the 

20 rectangular regions Al through A4 . The correlative values 
calculated for each of the rectangular regions Al through A4 
are designated as correlative values dOl, d02, d03, and d04, 
respectively. 

In the fourth embodiment, first, the correlative values 
25 dOl, d02, d03, and d04, which have been calculated in the case 
of the 2x2 arrangement, are each compared against the threshold 
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value Thl . For the regions that correspond to the rectangular 
regions within the reference patch PO and the patch PI, for which 
the correlative value is less than or equal to the threshold 
value Thl, the number of rectangular regions (in the 2x2 
5 arrangement) is determined to be the number of rectangular 
regions to be employed in generating the synthesized frame. On 
the other hand, for the rectangular regions, for which the 
correlative value exceeds the threshold value Thl, the number 
of rectangular regions is increased by one step, and the 

10 estimation of the correspondent relationship, the obtainment 
of the coordinate converted frame, and the calculation of the 
correlative value are performed again. The process is repeated, 
that is, the number of rectangular regions is increased and the 
correlative value is calculated, until the correlative value 

15 becomes less than or equal to the threshold value Thl for all 
of the regions within the reference patch PO and the patch PI. 
The number of rectangular regions that yields the correlative 
value less than or equal to the threshold value Thl is determined 
to be the number of rectangular regions to be employed in 

20 generating the synthesized frame, for regions within the 
reference patch PO and the patch PI that correspond to the 
rectangular region. 

For example, a case is considered in which the correlative 
values dOl and d02, of the rectangular regions Al and A2 in the 

25 2 x 2 arrangement, are less than or equal to the threshold value 
Thl, while the correlative values d03 and d04, of the 
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rectangular regions A3 and A4 exceed the threshold value Thl . 
In this case, the number of rectangular regions (in this case, 
2x2), is determined to be the number of rectangular regions 
to be employed in generating the synthesized frame, for the 
regions corresponding to the rectangular regions Al and A2 
within the reference patch PO and the patch PI. 

Then, the rectangular regions A3 and A4 are divided so 
that they have the same number of rectangular regions therein 
as in the case that rectangular regions are provided in the 
reference patch PO and the patch PI in a 4 x 4 arrangement. 
Correspondent relationships are estimated, coordinate 
converted frames are obtained, and correlative values are 
calculated only for the regions of the reference patch PO and 
the patch PI that correspond to the rectangular regions A3 and 
A4 . Next, the correlative values of each of the rectangular 
regions A31 through A34 and A41 through A44, which correspond 
to the rectangular regions A3 and A4 , are compared against the 
threshold value Thl. Assume that the correlative values of all 
of the rectangular regions A31 through A34 and A41 through A44 
are less than or equal to the threshold value Thl. The number 
of rectangular regions (that is, 4x4), is determined to be 
the number of rectangular regions to be employed in generating 
the synthesized frame, for the regions corresponding to the 
rectangular regions A31 through A34 and A41 through A44 within 
the reference patch PO and the patch PI . 

In this case, the number of rectangular regions, to be 



employed in generating the synthesized frame from the reference 
patch PO and the patch PI is 2 x 2 in the left region and 4 x 
4 in the right region, as shown in Figure 14. Note that in the 
case that the correlative values of the rectangular regions A31 
through A34 and A41 through A44 exceed the threshold value Thl, 
the numbers of rectangular regions therein are increased 
another step. Then, the estimation of correspondent 
relationships, the obtainment of coordinate converted frames 
and the calculation of correlative values are performed again. 
This is repeated until the correlative values become less than 
or equal to the threshold value Thl . 

Next, the operation of the fourth embodiment will be 
described. Figure 15 is a flow chart illustrating the processes 
performed by the moving image synthesizing apparatus of the 
fourth embodiment. First, moving image data MO is input to the 
sampling means 11 (step S21), and the reference frame FrN and 
the frame FrN+1 are sampled therefrom (step S22) . Next, a 
correspondent relationship between the reference frame FrN and 
the frame FrN+1 is estimated by the correspondent relationship 
estimating means 12 (step S23) . 

Then, the frame FrN+1 is converted into the coordinate 
space of the reference frame FrN by the coordinate converting 
means 13, based on the estimated correspondent relationships, 
to obtain the coordinate converted frame FrTO (step S24) . 
Thereafter, a plurality of correlative values, which represents 
the degrees of correlation among each rectangular region of the 



reference frame FrN and the coordinate converted frame FrTO, 
is calculated by the correlative value calculating means 14 
(step S25) . Whether the correlative value of one of the 
rectangular regions is less than or equal to the threshold value 
5 Thl is judged by the comparing means 15 (step S26) . 

In the case that the judgment in step S26 is negative, 
the number of rectangular regions within the rectangular region 
of the reference patch PO and the patch PI is increased by one 
step (step S27). Then, the process jumps to step S29, which 

10 will be described later. On the other hand, in the case that 
the judgment in step S26 is affirmative, the number of 
rectangular regions that yielded the correlative value for the 
rectangular region is determined to be that which is to be 
employed in generating the synthesized frame (step S28) . Next, 

15 whether the judgment of step S26 has been completed for all of 
the rectangular regions is judged (step S29) . In the case that 
the judgment in step S29 is negative, a different rectangular 
region is selected (step S30) , the process returns to step S26, 
and the steps thereafter are repeated. 

20 In the case that the judgment in step S29 is affirmative, 

whether the number of rectangular regions to be employed in 
generating the synthesized frame has been determined for all 
regions within the reference patch PO and the patch PI is judged 
(step S31) . In the case that the judgment in step S31 is 

25 negative, the process returns to step S23, and the steps 
thereafter are repeated. On the other hand, in the case that 
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the judgment in step S31 is affirmative, the synthesizing means 
16 generates the synthesized frame FrG from the reference frame 
FrN and the frame FrN+1, based on the correspondent 
relationships which were estimated for the determined number 
5 of rectangular regions (step S32), and the process ends. 

In this manner, the fourth embodiment determines the 
correspondent relationship to be employed in the generation of 
the synthesized frame FrG for each rectangular region. 
Accordingly, the synthesized frame FrG is generated in units 

10 of regions corresponding to rectangular regions into which the 
reference patch P0 and the patch PI have been divided. 
Therefore, the synthesized frame FrG is enabled to be generated 
employing the optimal number of rectangular regions for 
following the movement and/or deformation of a moving subject 

15 within the frames. Accordingly, a synthesized frame FrG having 
high image quality is enabled to be obtained, regardless of 
movement of a subject included in the frames. 

Note that in the fourth embodiment, the numbers of 
rectangular regions within the reference patch P0 and the patch 

20 Plare increased in a stepwise manner until the correlative 
values of all of the rectangular regions become less than or 
equal to the threshold value Thl . Alternatively, the number 
of rectangular regions within the reference patch P0 and the 
patch PI that yields a representative correlative value (for 

25 example, a mean correlative value, a median correlative value, 
a maximum correlative value, or a minimum correlative value) 
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less than or equal to a predetermined threshold value Th2, may 
be determined as that employed in generating the synthesized 
frame FrG. 

In the third and fourth embodiments, the numbers of 
5 rectangular regions within the reference patch PO and the patch 
Plare increased in a stepwise manner until the correlative 
values of all of the rectangular regions become less than or 
equal to the threshold value Thl . Alternatively, the number 
of rectangular regions within the reference patch PO and the 

10 patch PI that yields a correlative value less than or equal to 
the threshold value Thl in one of the rectangular regions may 
be determined to be that employed in generating the synthesized 
frame FrG. For example, if the correlative value of only one 
rectangular region (for example, region Al) is less than or 

15 equal to the threshold value Thl in the initial step (2x2 
arrangement) , then the 2x2 arrangement of rectangular regions 
is employed to generate the synthesized frame FrG. 

As a further alternative, a number of rectangular regions 
that yields a percentage of the rectangular regions within the 

20 second patch or the reference patch having correlative values 
less than or equal to the threshold value Thl, which is greater 
than or equal to a predetermined percentage (for example, 50%) , 
may be determined to be that to be employed in generating the 
synthesized frame FrG. For example, assume a case that the 

25 rectangular regions are provided in a 4 x 4 arrangement. If 
nine of the sixteen rectangular regions yield correlative 
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values less than or equal to the threshold value Thl, as 
indicated by the hatched regions in Figure 16, then the number 
of rectangular regions to be employed in generating the 
synthesized frame FrG may be determined to be sixteen. 
5 There is a high possibility that a main subject, such as 

a human face or a complex texture, is included in the vicinity 
of the center of the frames (hereinafter, referred to as "main 
region") . Therefore, as a still further alternative, a number 
of rectangular regions within the reference patch PO and the 

10 patch PI that yields correlative values less than or equal to 
the threshold value Thl in a main region may be determined to 
be that employed in generating the synthesized frame FrG. For 
example, assume a case that rectangular regions are provided 
in a 4 x 4 arrangement. If the four central rectangular regions 

15 are positioned at a portion of an image that includes a human 
face, as shown in Figure 17, then a number of rectangular regions 
that yields correlative values less than or equal to the 
threshold value Thl for the four central rectangular regions 
may be determined to be that employed in generating the 

20 synthesized frame FrG. Note that in this case, a mean of the 
correlative values of the rectangular regions positioned in the 
main region may be calculated, and compared against the 
threshold value Thl. 

In the case that the mean of the correlative values is 

25 calculated, the correlative values, which have been calculated 
for each rectangular region, may be weighted according to the 
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percentage that the main subject occupies within the 
rectangular region. Then, the mean correlative value dm may 
be calculated from the weighted correlative values, according 
to Formula (17) below. 
5 Formula (17) 

n 

wherein dOi: correlative values of each rectangular region 

Si: weighting coefficients that increase in 
proportion to the percentage of each rectangular region that 

10 the main subject occupies 

Further, in the fourth embodiment, the correlative values 
obtained of all of the rectangular regions at each step are 
compared against the threshold value Thl . However, local 
regions of the reference patch P0 and the patch PI may have 

15 extremely high correlative values. There are cases in which 
the correlative values of these local regions do not become less 
than or equal to the threshold value Thl, even if the number 
of rectangular regions within the patches P0 and PI are 
increased . For this reason, the correlative values of each 

20 rectangular region are compared against a predetermined value 
in the initial number of rectangular regions (provided in the 
2x2 arrangement) . In the case that the correlative value 
exceeds the predetermined value for a rectangular region, no 
further correlative values are calculated for that rectangular 

25 region. Thereby, the amount of calculation can be reduced, 
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thereby expediting the process. Note that the predetermined 
value may be several times the threshold value Thl . 

In the fourth embodiment, the threshold value Thl that 
the correlative values are compared against may be changed, 
5 according to the number of rectangular regions. Specifically, 
as shown in Figure 18, the threshold value Thl may be changed 
so that it decreases according to an increase in the number of 
rectangular regions until the number reaches a predetermined 
value (2 6 in this case), and then increases according to the 

10 increase in the number of rectangular regions. Thereby, the 
number of rectangular regions may be set appropriately. Note 
that if the number of rectangular regions becomes excessively 
large, each of the rectangular regions becomes too small. This 
will cause similar images to be represented in a plurality of 

15 rectangular images, which in turn may cause an inability to 
follow the movement and/or deformation of the subject. 
Therefore, the threshold value is increased after the number 
of rectangular regions exceeds the predetermined value. 

In the third and fourth embodiments, the- correlative 

2 0 values di are calculated for each of the luminance chrominance 
components Y, Cb, and Cr. However, a single set of correlative 
values di' may be calculated, by weighting correlative values 
diY for the luminance component, and correlative values diCb 
and diCr for the chrominance components with weighting 

25 coefficients al, bl, and cl, according to Formula (14). 

Alternatively, Euclid distances, weighted by weighting 
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coefficients a2, b2, and c2, among luminance components FrTi 
(x, y) and chrominance components FrTiCb (x, y) and FrTiCr (x, 
y) of a coordinate converted frame FrTi, and luminance 
components FrNY (x, y) , and chrominance components FrNCb and 
5 FrNCr, may be calculated for each of the pixels within the 
reference patch PO, according to Formula (15). The Euclid 
distances are designated as correlative values di" (x, y) for 
each pixel. When applied to the third embodiment, the mean 
value of the correlative values di" (x, y) for all of the pixels 

10 within the reference patch PO may be employed as the correlative 
value that determines the number of rectangular regions. When 
applied to the fourth embodiment, the mean value of the 
correlative values di" (x, y) for all of the pixels within each 
rectangular region may be employed as the correlative value that 

15 determines the number of rectangular regions. 

In the third and fourth embodiments described above, the 
synthesized frames FrG are generated from two frames FrN and 
FrN+1 . However, a synthesized frame FrG may be generated from 
three or more frames . For example, a case is considered in which 

20 a synthesized frame FrG is generated from T number of frames 
FrN+t' (wherein 0<t'<T-l) in the image synthesizing apparatus 
of the third embodiment. In this case, a plurality of 
correspondent relationships, between a reference frame FrN 
(=FrN+0) and each of a plurality of other frames FrN+t (wherein 

25 l<t<T-l), are estimated. Further, a plurality of coordinate 
converted frames are obtained, according to the plurality of 
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correspondent relationships. Then, correlative values are 
calculated between the reference frame FrN and each of the 
plurality of coordinate converted frames. The estimation of 
the correspondent relationships, the obtainment of the 
5 coordinate converted frames, and calculation of the correlative 
values are performed until the correlative value becomes less 
than or equal to the threshold value Thl . Next, a synthesized 
frame is generated, based on the correspondent relationship, 
which was estimated for the number of rectangular regions that 

10 yielded the correlative value less than or equal to the 
threshold value Thl. This synthesized frame is designated as 
an intermediate synthesized frame FrGt . Note that pixel values 
of the intermediate synthesized frame FrGt at integer 
coordinates of an integrated images are designated as FrGt (x^, 

15 y-). 

Intermediate synthesized frames FrGt are obtained for all 
of the other frames FrN+t. Finally, a synthesized frame FrG 
comprising pixel values fx", y A ) is generated, by adding 
corresponding pixels of the intermediate frames according to 

20 formula (16) . 

There are cases in which pixel values cannot be assigned 
to all of the integer coordinates of the integrated images. In 
these cases, the pixel values for the integer coordinates, for 
which no pixel values are assigned, may be calculated from 

25 assigned pixel values or from pixel values of the reference 
frame that correspond to the coordinates, for which no pixel 
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values are assigned. The calculation may be an interpolation 
calculation such as linear interpolation and spline 
interpolation . 

In the case that the synthesized frame FrG is generated 
5 from three or more frames, the intermediate synthesized frames 
may be generated in the same manner as that described in the 
fourth embodiment. That is, correlative values may be 
calculated for each of the rectangular regions within the 
patches, then the intermediate synthesized frames may be 

10 generated based on the correspondent relationships of each of 
the rectangular regions. 

Next, a fifth embodiment of the present invention will 
be described. Figure 19 is a block diagram illustrating the 
schematic construction of a moving image synthesizing apparatus 

15 according to the fifth embodiment of the present invention. The 
moving image synthesizing apparatus of the fifth embodiment 
comprises: a sampling means 21; a correspondent relationship 
estimating means 22; a coordinate converting means 23; a 
correlative value calculating means 24 ; a variation calculating 

2 0 means 25; a comparing means 2 6, and a synthesizing means 27, 
instead of the sampling means 1; the correspondent relationship 
estimating means 2; the coordinate converting means 3; the 
correlative value calculating means 4; the comparing means 5; 
and the synthesizing means 6 of the first embodiment. The 

25 coordinate converting means 23 coordinate converts frames other 
than a reference frame to a coordinate space of the reference 
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frame, based on the correspondent relationships estimated by 
the correspondent relationship estimating means 22. The 
coordinate converting means 23 thus obtains a plurality of 
coordinate converted frames FrTi . The correlative value 
5 calculating means 24 calculates a plurality of correlative 
values, which represent degrees of correlation among the 
coordinate converted frames FrTi and the reference frame. The 
variation calculating means 2 5 calculates the variation between 
two correlative values, which have been calculated for two 

10 consecutive steps in the stepwise increase of a number of 
rectangular regions within patches, as described later. The 
comparing means 2 6 compares the variation against a threshold 
value Th3 . The comparing means 2 6 causes the correspondent 
relationship estimating means 22, the coordinate converting 

15 means 23, the correlative value calculating means 24, and the 
variation calculating means 25 to increase the number of 
rectangular regions within patches, for which correspondent 
relationships are estimated, in a stepwise manner, to estimate 
correspondent relationships, to obtain coordinate converted 

20 frames FrTi, to calculate correlative values, and to calculate 
variations for each increased step of the number of rectangular 
regions, until the variation becomes less than or equal to the 
threshold value Th3. The synthesizing means 27 generates a 
synthesized frame FrG having a higher resolution than the 

25 sampled frames, by administering interpolation calculations on 
the reference frame and the other frame, based on the 
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correspondent relationships, which were estimated by the 
correspondent relationship estimating means 22 regarding the 
frames having the number of rectangular regions therein that 
yielded the variation less than or equal to the threshold value 
Th3. 

The sampling means 21 samples a plurality of frames from 
moving image data MO in the same manner as the sampling means 
1 of the first embodiment. In the fifth embodiment as well, 
the sampling means 21 samples two frames, FrN and FrN+1, from 
the moving image data MO . 

In the first embodiment, the correspondent relationship 
estimating means 2 estimated the first correspondent 
relationship, then estimated the second correspondent 
relationship after increasing the number of rectangular regions 
in a stepwise manner. In the fifth embodiment, the 
correspondent relationship estimating means 22 derives a first 
correspondent relationship, then estimates further 
correspondent relationships after increasing the number of 
rectangular regions, as will be described later. 

The coordinate converting means 23 obtains a coordinate 
converted frame FrTO by employing bilinear conversion, 
according to (5) and (6) , to coordinate convert the frame FrN+1 
to the coordinate space of the reference frame FrN, based on 
an estimated correspondent relationship. Note that in the 
following description, conversion and synthesis is performed 
only for the regions within a reference patch PO of the reference 
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frame FrN and a patch PI within the other frame FrN+1. 

The correlative value calculating means 2 4 calculates a 
correlative value dO, which represents the degree of 
correlation between the reference frame FrN and the coordinate 
5 converted frame FrTO, according to Formula (9) . 

In the case that the number of rectangular regions is an 
initial value (four rectangular regions provided in a 2 x 2 
arrangement in the present embodiment) , the correlative value 
calculating means 24 calculates the correlative value dO . Then, 

10 the number of rectangular regions is increased by one step (to 
sixteen rectangular regions provided in a 4 x 4 arrangement in 
the present embodiment) , and the correlative value calculating 
means 24 calculates a correlative value dl . The variation 
calculating means 25 calculates a variation AdO between the 

15 correlative value dO and the correlative value dl . 

Specifically, the correspondent relationship estimating 
means 22 increases the number of rectangular regions within the 
reference patch P0 and the patch PI from a 2 x 2 arrangement 
to a 4 x 4 arrangement, as shown in Figure 4A. The reference 

20 patch P0 and the patch PI are provided at appropriate positions 
in the reference frame FrN and the frame FrN+1, respectively. 
Then, the amount of movement and the amount of deformation of 
the patch PI with respect to the reference patch P0 is derived 
in the same manner as that described above. Thereafter, the 

25 correspondent relationships among the pixels of the reference 
patch P0 and the patch PI are estimated. Next, the coordinate 
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converting means 23 converts the frame FrN+1 to a coordinate 
space of the reference frame FrN, based on the correspondent 
relationships estimated by the correspondent relationship 
estimating means 22. The coordinate converting means 23 thus 
5 obtains a coordinate converted frame FrTl. The correlative 
value calculating means 24 calculates the correlative value dl, 
which represents the degree of correlation between the 
coordinate converted frame FrTl and the reference frame FrN, 
according to Formula (10) . Thereafter, the variation AdO is 
10 calculated as the absolute value of the difference between the 
correlative value dO and the correlative value dl, according 
to Formula (18) below. 

Formula (18) 

MO =| dO-d\\ 

15 Note that the variation AdO cannot be calculated by 

calculating only the correlative value dO . Therefore, the 
variation calculating means 25 controls the correspondent 
relationship estimating means 22, the coordinate converting 
means 23, and the correlative value calculating means 24 to 

20 increase the number of rectangular regions one step, and to 
calculate the correlative value dl . 

The comparing means 26 compares the variation AdO against 
the threshold value Th3 . The threshold value Th3 is 
predetermined and recorded in a memory (not shown) of the 

25 comparing means 26. In the case that the variation AdO is less 
than or equal to the threshold value Th3, the smaller of the 
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two numbers of rectangular regions that yielded the correlative 
values dO and dl, from which the variation AdO was calculated, 
is determined. This number of rectangular regions is 
designated as the number of rectangular regions for which the 
5 correspondent relationship, to be employed by the synthesizing 
means 27 to generate the synthesized frame FrG from the frames 
FrN and FrN+1, was estimated, as will be described later. 

On the other hand, in the case that the variation AdO 
exceeds the threshold value Th3, the comparing means 2 6 controls 

10 the correspondent relationship estimating means 22, the 
coordinate converting means 23, the correlative value 
calculating means 24, and the variation calculating means 25 
to increase the number of rectangular regions within the 
reference patch P0 and the patch PI one step in a stepwise manner, 

15 to estimate correspondent relationships, to obtain a coordinate 
converted frame, to calculate a correlative value, and to 
calculate a variation for the increased number of rectangular 
regions . 

Specifically, the correspondent relationship estimating 
20 means 22 increases the number of rectangular regions within the 
reference patch P0 and the patch PI from a 4 x 4 arrangement 
to an 8 x 8 arrangement, as shown in Figure 4B. The reference 
patch P0 and the patch PI are provided at appropriate positions 
in the reference frame FrN and the frame FrN+1 , respectively. 
25 Then, the amount of movement and the amount of deformation of 
the patch PI with respect to the reference patch P0 is derived 
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in the same manner as that described above. Thereafter, the 
correspondent relationships among the pixels of the reference 
patch PO and the patch PI are estimated. Next, the coordinate 
converting means 23 converts the frame FrN+1 to a coordinate 
5 space of the reference frame FrN, based on the correspondent 
relationships estimated by the correspondent relationship 
estimating means 22. The coordinate converting means 23 thus 
obtains a coordinate converted frame FrT2. The correlative 
value calculating means 24 calculates a correlative value d2, 

10 which represents the degree of correlation between the 
coordinate converted frame FrT2 and the reference frame FrN. 
Thereafter, the variation calculating means 25 calculates the 
absolute value of the difference between the correlative value 
dl and d2, as a variation Adl . 

15 Then, the comparing means 2 6 compares the variation Adl 

against the threshold value Th3 . In the case that the variation 
Adl exceeds the threshold value Th3, the number of rectangular 
regions within the reference patch PO and the patch PI is further 
increased, to 256 rectangular regions provided in a 16 x 16 

20 arrangement. Thereafter, the correspondent relationship 
estimating means 22, the coordinate converting means 23, the 
correlative value calculating means 24, and the variation 
calculating means 25 are caused to estimate correspondent 
relationships, to obtain coordinate converted frames, to 

25 calculate correlative values di + 1 (wherein -l<i) (hereinafter, 
the variable "i" will be employed for generalization) , and to 
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calculate variations for each stepwise increase in the number 
of rectangular regions, until the variation Adi between the 
correlative values di and di+1 becomes less than or equal to 
the threshold value Th3 . Note that in the fifth embodiment, 
5 the number of rectangular regions within the reference patch 
PO and the patch pi of the i-th step is 2 i+1 x 2 i+1 . The coordinate 
converted frame, obtained by the number of rectangular regions 
of the i-th step, is designated as coordinate converted frame 
FrTi. The correlative value, calculated for the patches having 

10 therein the number of rectangular regions of the i-th step, is 
designated as correlative value di . The smaller of the two 
numbers of rectangular regions within the reference patch PO 
and the patch PI that yield the variation Adi which is less than 
or equal to the threshold value Th3 is determined. This number 

15 of rectangular regions is designated as the number of 
rectangular regions to be employed to generate the synthesized 
frame . 

Note that if the number of rectangular regions becomes 
excessively large, each of the rectangular regions becomes too 
20 small. This will cause similar images to be represented in a 
plurality of rectangular images, which in turn may cause an 
inability for the movement and/or deformation of the second 
patch to follow the movement and/or deformation of the subject. 
Therefore, it is preferable that an upper limit is set for the 
25 number of rectangular regions. 

The synthesizing means 27 administers interpolation 
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calculations on the reference frame FrN and the frame FrN+1 , 
based on the correspondent relationship, which was estimated 
for the number of rectangular regions designated to be employed 
in the generation of the synthesized frame, and generates the 
5 synthesized frame FrG in the same manner as the synthesizing 
means 6 of the first embodiment. 

Note that in the fifth embodiment, synthesized frames FrG 
are obtained for the luminance component Y and the chrominance 
components Cb and Cr, as in the first embodiment. A final 

10 synthesized frame is obtained by synthesizing the synthesized 
frame FrG(Y) derived from the luminance component Y, a 
synthesized frame FrG(Cb) derived from the chrominance 
component Cb, and a synthesized frame FrG(Cr) , derived from the 
chrominance component Cr . 

15 Next, the operation of the fifth embodiment will be 

described. Figure 20 is a flow chart illustrating the processes 
performed by the moving image synthesizing apparatus of the 
fifth embodiment. First, moving image data MO is input to the 
sampling means 21 (step S41), and the reference frame FrN and 

20 the frame FrN+1 are sampled therefrom (step S42). Next, a 
correspondent relationship between the reference frame FrN and 
the frame FrN+1 is estimated by the correspondent relationship 
estimating means 22 (step S43) . 

Then, the frame FrN+1 is converted into the coordinate 

25 space of the reference frame FrN by the coordinate converting 
means 23, based on the estimated correspondent relationships, 
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to obtain the coordinate converted frame FrTi (step S44) . 
Thereafter , the correlative value di, which represents the 
degree of correlation between the reference frame FrN and the 
coordinate converted frame FrTO, is calculated by the 
5 correlative value calculating means 24 (step S45) . 

Thereafter, whether the correlative value has been 
calculated for the reference patch PO and the patch PI of the 
Oth step, that is, the initial number of rectangular regions, 
is judged (step S4 6) . In the case that the judgment in step 

10 S46 is affirmative, the number of rectangular regions within 
the reference patch PO and the patch PI is increased one step 
(step S47) . Then, the process returns to step S43, and steps 
S43 through S45 are repeated with respect to the reference patch 
PO and the patch PI having the increased number of rectangular 

15 regions therein. Thereby, a correlative value di+1, which 
represents the degree of correlation between the reference 
frame FrN and the coordinate converted frame FrTO, is 
calculated. 

After the correlative value di+1 is calculated with 
20 respect to the increased number of rectangular regions, the 
judgment in step S4 6 becomes negative. The variation Adi 
between the correlative value di, which was calculated for the 
number of rectangular regions of the previous step, and the 
correlative value di+1, which was calculated for the current 
25 number of rectangular regions, is calculated (step S48) . Then, 
whether the variation Adi is less than or equal to the threshold 
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value Th3 is judged by the comparing means 26 (step S49) . 

In the case that the judgment in step S49 is negative, 
the number of rectangular regions within the reference patch 
PO and the patch PI is increased by another step (step S47) . 
5 Then, the process returns to step S43, and steps S43 through 
S49 are repeated with respect to the reference patch PO and the 
patch PI having the increased number of rectangular regions 
therein. 

In the case that the judgment in step S4 9 is affirmative, 

10 the smaller of the two numbers of rectangular regions that 
yielded the correlative value di, of the correlative values di 
and di+1, for which the variation Adi was calculated, is 
determined to be that employed in generating the synthesized 
frame (step S50) . Then, the synthesizing means 27 generates 

15 the synthesized frame FrG from the reference frame FrN and the 
frame FrN+1, based on the correspondent relationship which was 
estimated for the determined number of rectangular regions 
(step S51), and the process ends. 

In this manner, the synthesized frame FrG is generated 

20 from the reference frame FrN and the frame FrN+1, based on the 
correspondent relationship which was estimated for the number 
of rectangular regions that yielded the variation less than or 
equal to the threshold value Th3, in the fifth embodiment. 
Therefore, the synthesized frame FrG is enabled to be generated 

25 employing the optimal number of rectangular regions for 
following the movement and/or deformation of a moving subject 
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within the frames. Accordingly, a synthesized frame FrG having 
high image quality is enabled to be obtained, regardless of 
movement of a subject included in the frames. 

Next, the sixth embodiment of the present invention will 
5 be described. In the fifth embodiment described above, the 
synthesized frame FrG is generated based on the correspondent 
relationship, estimated for the number of rectangular regions 
that yields the variation less than or equal to the threshold 
value Th3, of the entirety of the images in the patches within 

10 the reference frame FrN and the frame FrN+1. In the sixth 
embodiment, variations are calculated for each rectangular 
region within the reference patch P0 and the patch 1. Each of 
the variations are compared against the threshold value Th3, 
and a synthesized frame FrG is generated after determining the 

15 number of rectangular regions to be employed in generating the 
synthesized frame FrG for each of the rectangular regions . Note 
that in the sixth embodiment, Formula (9) is employed to 
calculate the sums of the absolute values of the differences 
between the pixel values FrN (x, y) and FrTO (x, y) , for 

20 corresponding pixels within the reference frame FrN and the 
coordinate converted frame FrTO, divided by the number of pixels 
m, with respect to all of the pixels within each of the 
rectangular regions . The mean value of the quotients, that is, 
the absolute values of the differences of each pair of 

25 corresponding pixels, is calculated as the correlative value 
of each rectangular region. 
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Note that the calculation of the correlative values for 
each of the rectangular regions at each step is performed in 
the same manner as in the fourth embodiment, therefore a 
detailed description will be omitted here. Here, the 
5 correlative values calculated for each of the rectangular 
regions Al through A4 are designated as correlative values dOl, 
d02, d03, and d04, respectively. 

In the sixth embodiment, sums of the correlative values 
of the rectangular regions All through A14, A21 through A24, 
10 A31 through A34, and A41 through A44 are calculated for the 
sixteen rectangular regions provided in the 4x4 arrangement, 
in order to calculate variations. The summed correlative 
values are designated as correlative values dll, dl2, dl3 and 
dl4. 

15 The absolute values of the differences among correlative 

values of corresponding rectangular regions are calculated as 
the variations . That is, variations AdOl , Ad02, Ad03, andAd04, 
corresponding to the rectangular regions Al through A4, 
provided in the 2x2 arrangement, are calculated according to 
2 0 Formulas (19) through (22) below. 
Formula (19) 
Ad0l=\ d0l-dll\ 
Formula (20) 
AdOl =| d02-dl2\ 
25 Formula (21) 

Ad03 =\ d03-d\3\ 
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Formula (22) 
Ad04 =| d04-dl4\ 

Then, the variations AdOl, Ad02, Ad03, and Ad04 are compared 
against the threshold value Th3 . For the regions that 
5 correspond to the rectangular regions within the reference 
patch PO and the patch PI, for which the variation is less than 
or equal to the threshold value Th3, the smaller of the two 
numbers of rectangular regions (that is, the 2x2 arrangement) 
is determined to be the number of rectangular regions to be 

10 employed in generating the synthesized frame . On the other hand, 
for the rectangular regions, for which the variation exceeds 
the threshold value Th3, the number of rectangular regions is 
increased by one step, and the estimation of the correspondent 
relationship, the obtainment of the coordinate converted frame, 

15 the calculation of the correlative value, and the calculation 
of the variation are performed again. The process is repeated, 
that is, the number of rectangular regions is increased, the 
correlative value is calculated, and the variation is 
calculated, until the variation becomes less than or equal to 

20 the threshold value Th3 for all of the regions within the 
reference patch PO and the patch PI . The smaller of the two 
numbers of rectangular regions that yield the variation less 
than or equal to the threshold value Th3 is determined to be 
the number of rectangular regions to be employed in generating 

25 the synthesized frame, for regions within the reference patch 
PO and the patch PI that correspond to the rectangular region. 
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For example, a case is considered in which the variations 
AdOl and Ad02, of the rectangular regions Al and A2 in the 2 
x 2 arrangement, are less than or equal to the threshold value 
Th3, while the variations Ad03 and Ad04, of the rectangular 
5 regions A3 and A4 exceed the threshold value Th3 . In this case, 
the smaller of the two numbers of rectangular regions (in this 
case, 2x2), is determined to be the number of rectangular 
regions to be employed in generating the synthesized frame, for 
the regions corresponding to the rectangular regions Al and A2 

10 within the reference patch P0 and the patch PI. 

On the other hand, the rectangular regions A31 through 
A34 and A41 through 44, corresponding to the rectangular regions 
A3 and A4, are further divided so that they have the same number 
of rectangular regions therein as in the case that rectangular 

15 regions are provided in the reference patch P0 and the patch 
PI in a 8 x 8 arrangement. Correspondent relationships are 
estimated, coordinate converted frames are obtained, and 
correlative values are calculated only for the regions of the 
reference patch P0 and the patch PI that correspond to the 

20 rectangular regions A31 through A34 and A41 through A44. Next, 
the correlative values of the four rectangular regions that 
correspond to each of the rectangular regions A31 through A34 
and A41 through A44 are summed. The summed correlative values 
are designated as the correlative values of the regions that 

25 correspond to the rectangular regions A31 through A34 and A41 
through A44. The variations among the correlative values of 
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the rectangular regions A31 through A34 and A41 through A44 and 
the correlative values of the rectangular regions, provided in 
the 8x8 arrangement, which correspond to the rectangular 
regions A31 through A34 and A41 through A4 4, are calculated. 
5 The variations are compared against the threshold value Th3 . 

Assume that the variations of all of the rectangular 
regions A31 through A34 and A41 through A44 are less than or 
equal to the threshold value Th3 . The smaller of the two numbers 
of rectangular regions (that is, sixteen regions provided in 

10 the 4x4 arrangement) , is determined to be the number of 
rectangular regions to be employed in generating the 
synthesized frame, for the regions corresponding to the 
rectangular regions A31 through A34 and A41 through A44 within 
the reference patch P0 and the patch PI. 

15 In this case, the number of rectangular regions, to be 

employed in generating the synthesized frame from the reference 
patch P0 and the patch PI, is 2 x 2 in the left region and 4 
x 4 in the right region, as shown in Figure 14. Note that in 
the case that the variations of the rectangular regions A31 

20 through A34 and A41 through A44 exceed the threshold value Th3, 
the numbers of rectangular regions therein are increased 
another step. Then, the estimation of correspondent 
relationships, the obtainment of coordinate converted frames, 
the calculation of correlative values and the calculation of 

25 the variations are performed again. This is repeated until the 
variations of all of the regions within the reference patch P0 
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and the patch PI become less than or equal to the threshold value 
Th3. 

Next, the operation of the sixth embodiment will be 
described. Figure 21 is a flow chart illustrating the processes 
5 performed by the moving image synthesizing apparatus of the 
sixth embodiment. First, moving image data MO is input to the 
sampling means 21 (step S61), and the reference frame FrN and 
the frame FrN+1 are sampled therefrom (step S62) . Next, a 
correspondent relationship between the reference frame FrN and 

10 the frame FrN+1 is estimated by the correspondent relationship 
estimating means 22 (step S63) . 

Then, the frame FrN+1 is converted into the coordinate 
space of the reference frame FrN by the coordinate converting 
means 23, based on the estimated correspondent relationships, 

15 to obtain the coordinate converted frame FrTi (step S64) . 
Thereafter, a plurality of correlative values, which represents 
the degrees of correlation among each rectangular region of the 
reference frame FrN and the coordinate converted frame FrTO, 
is calculated by the correlative value calculating means 24 

20 (step S65) . 

Thereafter, whether the correlative values have been 
calculated for the reference patch P0 and the patch PI of the 
0 th step, that is, the initial number of rectangular regions, 
is judged (step S66) . In the case that the judgment in step 

25 S66 is affirmative, the number of rectangular regions within 
the reference patch P0 and the patch PI is increased one step 
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(step S67) . Then, the process returns to step S63, and steps 
S63 through S65 are repeated with respect to the reference patch 
PO and the patch PI having the increased number of rectangular 
regions therein. Thereby, correlative values, which represent 
5 the degrees of correlation among each rectangular region of the 
reference frame FrN and the coordinate converted frame FrTO, 
are calculated. 

After the correlative values are calculated with respect 
to the increased number of rectangular regions, the judgment 

10 in step S66 becomes negative. The variations among the 
correlative values, which were calculated for the number of 
rectangular regions of the previous step, and the correlative 
values, which were calculated for the current number of 
rectangular regions, are calculated (step S68) . Then, whether 

15 the variations of each rectangular region are less than or equal 
to the threshold value Th3 is judged by the comparing means 2 6 
(step S69) . 

In the case that the judgment in step S69 is negative, 
the number of rectangular regions within the reference patch 

20 PO and the patch PI is increased by another step (step S70) . 
Then, the process jumps to step S72, which will be described 
later. On the other hand, in the case that the judgment in step 
S69 is affirmative, the smaller of the two numbers of 
rectangular regions that yielded the variation for the 

25 rectangular region is determined to be that which is to be 
employed in generating the synthesized frame (step S71) . Next, 
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whether the judgment of step S69 has been completed for all of 
the rectangular regions is judged (step S72) . In the case that 
the judgment in step S72 is negative, a different rectangular 
region is selected (step S73) , the process returns to step S69, 
5 and the steps thereafter are repeated. 

In the case that the judgment in step S72 is affirmative, 
whether the number of rectangular regions to be employed in 
generating the synthesized frame has been determined for all 
regions within the reference patch PO and the patch PI is judged 

10 (step S74) . In the case that the judgment in step S74 is 
negative, the process returns to step S73, and the steps 
thereafter are repeated regarding the rectangular regions, for 
which the number of rectangular regions to be employed in 
generating the synthesized frame has not been determined. On 

15 the other hand, in the case that the judgment in step S74 is 
affirmative, the synthesizing means 27 generates the 
synthesized frame FrG from the reference frame FrN and the frame 
FrN+1, based on the correspondent relationships which were 
estimated for the determined number of rectangular regions 

20 (step S75) , and the process ends. 

In this manner, the sixth embodiment determines the 
correspondent relationship to be employed in the generation of 
the synthesized frame FrG for each rectangular region. 
Accordingly, the synthesized frame FrG is generated in units 

25 of regions corresponding to rectangular regions into which the 
reference patch PO and the patch PI have been divided. 
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Therefore, the synthesized frame FrG is enabled to be generated 
employing the optimal number of rectangular regions for 
following the movement and/or deformation of a moving subject 
within the frames. Accordingly, a synthesized frame FrG having 
5 high image quality is enabled to be obtained, regardless of 
movement of a subject included in the frames. 

Note that in the sixth embodiment, the numbers of 
rectangular regions within the reference patch PO and the patch 
Plare increased in a stepwise manner until the variations of 

10 all of the rectangular regions become less than or equal to the 
threshold value Th3 . Alternatively, the number of rectangular 
regions within the reference patch PO and the patch PI that 
yields a representative variation (for example, a mean 
variation, a median variation, a maximum variation, or a minimum 

15 variation) less than or equal to a predetermined threshold value 
Th4, may be determined as that employed in generating the 
synthesized frame FrG. 

In the sixth embodiment, the numbers of rectangular 
regions within the reference patch PO and the patch Plare 

20 increased in a stepwise manner until the variations of all of 
the rectangular regions become less than or equal to the 
threshold value Th3 . Alternatively, the number of rectangular 
regions within the reference patch PO and the patch PI that 
yields a variation less than or equal to the threshold value 

25 Th3 in one of the rectangular regions may be determined to be 
that employed in generating the synthesized frame FrG. For 
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example, if the variation of only one rectangular region (for 
example, region Al) is less than or equal to the threshold value 
Th3 in the initial step (2x2 arrangement) , then the 2x2 
arrangement of rectangular regions is employed to generate the 
5 synthesized frame FrG. 

As a further alternative, a number of rectangular regions 
that yields a percentage of the rectangular regions within the 
second patch or the reference patch having variations less than 
or equal to the threshold value Th3, which is greater than or 

10 equal to a predetermined percentage (for example, 50%) , may be 
determined to be that to be employed in generating the 
synthesized frame FrG. For example, assume a case that the 
rectangular regions are provided in a 4 x 4 arrangement. If 
nine of the sixteen rectangular regions yield variations less 

15 than or equal to the threshold value Th3, as indicated by the 
hatched regions in Figure 16, then the number of rectangular 
regions to be employed in generating the synthesized frame FrG 
may be determined to be sixteen. 

There is a high possibility that a main subject, such as 

20 a human face or a complex texture, is included in the vicinity 
of the center of the frames (hereinafter, referred to as "main 
region") . Therefore, as a still further alternative, a number 
of rectangular regions within the reference patch P0 and the 
patch PI that yields variations less than or equal to the 

25 threshold value Th3 in a main region may be determined to be 
that employed in generating the synthesized frame FrG. For 
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example, assume a case that rectangular regions are provided 
in a 4 x 4 arrangement. If the four central rectangular regions 
are positioned at a portion of an image that includes a human 
face, as shown in Figure 17, then a number of rectangular regions 
5 that yields variations less than or equal to the threshold value 
Th3 for the four central rectangular regions may be determined 
to be that employed in generating the synthesized frame FrG. 
Note that in this case, a mean of the variations of the 
rectangular regions positioned in the main region may be 

10 calculated, and compared against the threshold value Thl . 

In the case that the mean of the variations is calculated, 
the variations, which have been calculated for each rectangular 
region, may be weighted according to the percentage that the 
main subject occupies within the rectangular region. Then, the 

15 mean variation Adm may be calculated from the weighted 
variations, according to Formula (23) below. 
Formula (23) 

TSiAdOi 

Adm = ^ 

n 

wherein AdOi: correlative values of each rectangular 

20 region 

Si: weighting coefficients that increase in 
proportion to the percentage of each rectangular region that 
the main subject occupies 

Further, in the sixth embodiment, the variations obtained 
25 of all of the rectangular regions at each step are compared 
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against the threshold value Th3 . However, local regions of the 
reference patch PO and the patch PI may have extremely high 
variations. There are cases in which the variations of these 
local regions do not become less than or equal to the threshold 
5 value Th3, even if the number of rectangular regions within the 
patches PO and PI are increased. For this reason, the 
variations of each rectangular region are compared against a 
predetermined value in the initial number of rectangular 
regions (provided in the 2x2 arrangement) . In the case that 

10 the variation exceeds the predetermined value for a rectangular 
region, no further variations are calculated for that 
rectangular region. Thereby, the amount of calculation can be 
reduced, thereby expediting the process. Note that the 
predetermined value may be several times the threshold value 

15 Th3. 

In the fifth and sixth embodiments, the threshold value 
Th3 that the variations are compared against may be changed, 
according to the number of rectangular regions. Specifically, 
as shown in Figure 22, the threshold value Th3 may be changed 

20 so that it remains constant regardless of an increase in the 
number of rectangular regions until the number reaches a 
predetermined value (2 8 in this case) , and then increases 
according to the increase in the number of rectangular regions. 
Thereby, the number of rectangular regions may be set 

25 appropriately. Note that if the number of rectangular regions 
becomes excessively large, each of the rectangular regions 
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becomes too small. This will cause similar images to be 
represented in a plurality of rectangular images, which in turn 
may cause an inability to follow the movement and/or deformation 
of the subject. Therefore, the threshold value is increased 
after the number of rectangular regions exceeds the 
predetermined value. 

Note that in the first through sixth embodiments, the 
synthesized frame FrG is obtained for each of the luminance and 
chrominance components Y, Cb, and Cr of the reference frame FrM 
and the frame FrN+1. However, the synthesized frame FrG may 
be obtained only for the luminance component Y. Regarding the 
chrominance components Cb and Cr, synthesized frames therefor 
may be obtained by performing linear interpolations on the 
chrominance components Cb and Cr of the reference frame FrN. 

In the case that the frames FrN and FrN+1 comprise RGB 
color data, the synthesized frame FrG may be generated by 
processing each of the components R, G, and B. 

In the fifth and sixth embodiments, the correlative 
values di are calculated for each of the luminance chrominance 
components Y, Cb, and Cr. However, a single set of correlative 
values di' may be calculated, by weighting correlative values 
diY for the luminance component, and correlative values diCb 
and diCr for the chrominance components with weighting 
coefficients al, bl, and cl, according to Formula (14) . Then, 
variations Adi' may be calculated from the correlative values 
di' and correlative values di+1', which have been calculated 
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for the next step in the number of rectangular regions. 

Alternatively, Euclid distances, weighted by weighting 
coefficients a2, b2, and c2, among luminance components FrTi 
(x, y) and chrominance components FrTiCb (x, y) and FrTiCr (x, 
5 y) of a coordinate converted frame FrTi, and luminance 
components FrNY (x, y) , and chrominance components FrNCb and 
FrNCr, may be calculated for each of the pixels within the 
reference patch PO, according to Formula (15) . The Euclid 
distances are designated as correlative values di" (x, y) for 

10 each pixel. When applied to the fifth embodiment, the mean 
value of the correlative values di" (x, y) for all of the pixels 
within the reference patch PO may be employed as the correlative 
value to be used in calculating the variation that determines 
the number of rectangular regions. When applied to the sixth 

15 embodiment, the mean value of the correlative values di" (x, 
y) for all of the pixels within each rectangular region may be 
employed as the correlative value to be used in calculating the 
variation that determines the number of rectangular regions. 

In the fifth and sixth embodiments described above, the 

20 synthesized frames FrG are generated from two frames FrN and 
FrN+1. However, a synthesized frame FrG may be generated from 
three or more frames . For example, a case is considered in which 
a synthesized frame FrG is generated from T number of frames 
FrN+t' (wherein 0<t'<T-l) in the image synthesizing apparatus 

25 of the third embodiment." In this case, a plurality of 
correspondent relationships, between a reference frame FrN 
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(=FrN+0) and each of a plurality of other frames FrN+t (wherein 
l<t<T-l), are estimated. Further, a plurality of coordinate 
converted frames are obtained, according to the plurality of 
correspondent relationships. Then, correlative values are 
5 calculated between the reference frame FrN and each of the 
plurality of coordinate converted frames. Then, variations 
among the correlative values, obtained for two consecutive 
stepwise increases in the number of rectangular regions, are 
calculated. The estimation of the correspondent relationships, 

10 the obtainment of the coordinate converted frames, the 
calculation of the correlative values, and the calculation of 
the variations are performed until the variation becomes less 
than or equal to the threshold value Th3 . Next, a synthesized 
frame is generated, based on the correspondent relationship, 

15 which was estimated for the number of rectangular regions that 
yielded the variation less than or equal to the threshold value 
Th3 . This synthesized frame is designated as an intermediate 
synthesized frame FrGt . Note that pixel values of the 
intermediate synthesized frame FrGt at integer coordinates of 

20 an integrated images are designated as FrGt (x^, y A ). 

Intermediate synthesized frames FrGt are obtained for all 
of the other frames FrN+t. Finally, a synthesized frame FrG 
comprising pixel values (x^, y") is generated, by adding 
corresponding pixels of the intermediate frames according to 

2 5 formula (16) . 

There are cases in which pixel values cannot be assigned 
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to all of the integer coordinates of the integrated images. In 
these cases, the pixel values for the integer coordinates, for 
which no pixel values are assigned, may be calculated from 
assigned pixel values or from pixel values of the reference 
5 frame that correspond to the coordinates, for which no pixel 
values are assigned. The calculation may be an interpolation 
calculation such as linear interpolation and spline 
interpolation . 

In the case that the synthesized frame FrG is generated 

10 from three or more frames, the intermediate synthesized frames 
may be generated in the same manner as that described in the 
fourth embodiment. That is, correlative values may be 
calculated for each of the rectangular regions within the 
patches, then the intermediate synthesized frames may be 

15 generated based on the correspondent relationships of each of 
the rectangular regions. 

In the fifth and sixth embodiments, the variations Adi 
are calculated from two correlative values di and di+1. The 
number of rectangular regions, the correspondent relationship 

20 of which is to be employed in the generation of a synthesized 
frame, is determined to be the smaller of the two numbers of 
rectangular regions, for which the correlative value di was 
calculated. However, the larger number of rectangular regions, 
for which the correlative value di + 1 was calculated, may be the 

25 number of rectangular region, the correspondent relationship 
of which is to be employed in the generation of the synthesized 
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frame . 
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